Utilizing vehicle window shading to improve quality of augmented reality video

ABSTRACT

A light shading module is integrated with a vehicle window and is configured to be in different states that involve different values of Visible Light Transmittance (VLT) of the vehicle window. A camera fixed to the vehicle takes video of the outside environment. A computer generates a video see-through (VST) of the outside environment, which is presented to the vehicle&#39;s occupant on an augmented reality device (ARD). In order to improve the image quality, the ARD operates according to different modes involving varying intensities, depending on the VLT of the window.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application is a Continuation of U.S. application Ser. No.15/335,404, filed Oct. 26, 2016, which claims priority to U.S.Provisional Patent Application No. 62/246,187, filed on 26 Oct. 2015,and U.S. Provisional Patent Application No. 62/369,127, filed on 31 Jul.2016.

ACKNOWLEDGMENTS

Gil Thieberger would like to thank his holy and beloved teacher, LamaDvora-hla, for her extraordinary teachings and manifestation of wisdom,love, compassion and morality, and for her endless efforts, support, andskills in guiding him and others on their paths to freedom and ultimatehappiness. Gil would also like to thank his beloved parents for raisinghim exactly as they did.

BACKGROUND

It has become more and more common for vehicle occupants to engage invarious work- or entertainment-related activities. The activitiestypically involve various forms of displays which the occupants viewinstead of looking out of the vehicle, and can offer a productive orentertaining way to pass the time spent traveling. The quality of theviewing experience can be influenced by the amount of ambient light thatpenetrates the vehicle.

US patent application num. 20150261219 describes an autonomous modecontroller configured to control the operation of shaded vehiclewindows. However, the operation is unrelated to watching video. Thus,there is a need to be able to control ambient light levels in a vehiclein a way related to consumption of video content while in the vehicle.

SUMMARY

Some aspects of this disclosure involve a system that utilizes windowshading of a vehicle window in order to improve the quality of videoviewed by an occupant of the vehicle who wears a head-mounted display(HMD).

In one embodiment, an autonomous on-road vehicle includes a systemconfigured to enable an HMD to cooperate with a window light shadingmodule. This embodiment involves a light shading module, a camera, acomputer, and the HMD. The light shading module is integrated with avehicle window and is configured to be in at least first and secondstates. In the first state the Visible Light Transmittance (VLT) of thevehicle window is above 10% of ambient light entering through thewindow, in the second state the VLT of the vehicle window is below 50%of ambient light entering through the window, and the VLT of the vehiclewindow in the first state is higher than the VLT of the vehicle windowin the second state. The camera is fixed to the vehicle and configuredto take video of the outside environment. The computer is configured togenerate, based on the video, a video see-through (VST) that representsthe outside environment from a point of view of an occupant looking tothe outside environment through at least a portion of the vehiclewindow. The HMD comprises an optical see-through component and a displaycomponent; the HMD is configured to operate according to a first mode ofoperation when the occupant looks at the direction of the vehicle windowand the light shading module is in the first state, and to operateaccording to a second mode of operation when the occupant looks at thedirection of the vehicle window and the light shading module is in thesecond state. Wherein the total intensity of the VST light, emitted bythe display component and reaching the occupant's eyes, is higher in thesecond mode than in the first mode.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are herein described by way of example only, withreference to the accompanying drawings. No attempt is made to showstructural details of the embodiments in more detail than is necessaryfor a fundamental understanding of the embodiments. In the drawings:

FIG. 1 is a schematic illustration of components of a system configuredto combine video see-through (VST) with video-unrelated-to-the-VST(VUR);

FIG. 2 illustrates an HMD tracking module that measures the position ofthe HMD relative to the compartment;

FIG. 3 illustrates a vehicle in which an occupant wears an HMD;

FIG. 4 illustrates an occupant wearing an HMD and viewing large VUR andsmaller VST;

FIG. 5a illustrates how the VST moves to the upper left when theoccupant looks to the bottom right;

FIG. 5b illustrates how the VST moves to the bottom right when theoccupant looks to the upper left;

FIG. 6 illustrates HMD-video that includes both a non-transparent VSTand video that shows the hands of the occupant and the interior of thecompartment;

FIG. 7 illustrates HMD-video that includes both a partially transparentVST and video that shows the hands of the occupant and the interior ofthe compartment;

FIG. 8 illustrates HMD-video that includes a VST and partiallytransparent video that shows the hands of the occupant and the interiorof the compartment;

FIG. 9a illustrates HMD-video that includes a VUR in full FOV, a firstwindow comprising compartment-video (CV) and a second smaller windowcomprising the VST;

FIG. 9b illustrates HMD-video that includes VUR in full FOV, a firstwindow comprising the CV and a second partially transparent smallerwindow comprising the VST;

FIG. 10a illustrates HMD-video that includes VUR in full FOV, a firstwindow comprising VST and a second smaller window comprising zoom out ofthe CV;

FIG. 10b illustrates HMD-video that includes VUR and a partiallytransparent CV;

FIG. 11a illustrates a FOV of a vehicle occupant when the occupant wearsan HMD that presents HMD-video;

FIG. 11b illustrates a FOV of a vehicle occupant when the vehicleoccupant does not wear an HMD that presents the video, such as whenwatching an autostereoscopic display;

FIG. 11c illustrates FOV of a 3D camera that is able to capture sharpimages from different focal lengths;

FIG. 12 is a schematic illustration of components of a system configuredto enable an HMD to cooperate with a window light shading module;

FIG. 13a illustrates a first mode for a shading module where an occupantsees the outside environment through the optical see-through component;

FIG. 13b illustrates a second mode for a shading module where theoccupant sees the outside environment through a VST;

FIG. 14 illustrates a VST over a curtain;

FIG. 15 illustrates a light shading module that is unfurled on theinside of the compartment;

FIG. 16 illustrates a light shading module that is unfurled on theoutside of the compartment;

FIG. 17 is a schematic illustration of components of a video system thatmay be used to increase awareness of an occupant of a vehicle regardingan imminent SDRS;

FIG. 18a illustrates presenting VUR to an occupant when there is noindication that an SDRS event is imminent;

FIG. 18b illustrates presenting VST responsive to receiving anindication that an SDRS event is imminent (a pothole);

FIG. 18c illustrates presenting VST responsive to receiving anindication that an SDRS event is imminent (a sharp turn);

FIG. 19a illustrates presenting VUR and VST when there is no indicationthat an SDRS event is imminent;

FIG. 19b illustrates presenting a larger VST responsive to receiving anindication that an SDRS event is imminent (a road bump);

FIG. 19c illustrates presenting a partially transparent VST responsiveto receiving an indication that an SDRS event is imminent;

FIG. 20a illustrates a smart glass shading module when there is noindication that an SDRS event is imminent;

FIG. 20b illustrates the smart glass shading module when there is anindication that an SDRS event is imminent;

FIG. 21a and FIG. 21b illustrate vehicles with an SAEDP in theircompartment were an occupant uses an HMD to receive a representation ofthe outside environment;

FIG. 22 illustrates a vehicle with an SAEDP in the vehicle's compartmentwith displays;

FIG. 23 illustrates how an SAEDP protects the occupant in a sidecollision;

FIG. 24a and FIG. 24b illustrate a vehicle with a motor configured tomove a nontransparent SAEDP to cover a side window;

FIG. 25a and FIG. 25b illustrate a vehicle with a motor configured tomove a front SAEDP between first and second states;

FIG. 26a illustrates an SAEDP mounted to the front of a vehicle at eyelevel of an occupant of the vehicle;

FIG. 26b illustrates an outer SAEDP that includes two air bags;

FIG. 27a and FIG. 27b illustrate a motorized external SAEDP that canmove between first and second states multiple times;

FIG. 28 illustrates a vehicle compartment in which an occupant may laydown;

FIG. 29 illustrates a vehicle with a front mirror;

FIG. 30a illustrates one example of a configuration of a plurality ofbeams located in the left side of the vehicle;

FIG. 30b illustrates one example of a configuration of a plurality ofbeams located in the front of the vehicle; and

FIG. 31a and FIG. 31b are schematic illustrations of computers able torealize one or more of the embodiments discussed herein.

DETAILED DESCRIPTION

The following are definitions of various terms that may be used todescribe one or more of the embodiments in this disclosure.

The terms “autonomous on-road vehicle” and “autonomous on-road mannedvehicle” refer to cars and motorcycles designed to drive on publicroadways utilizing automated driving of level 3 and above according toSAE International standard J3016 “Taxonomy and Definitions for TermsRelated to On-Road Motor Vehicle Automated Driving Systems”. Forexample, the autonomous on-road vehicle may be a level 3 vehicle, inwhich within known, limited environments, drivers can safely turn theirattention away from driving tasks; the autonomous on-road vehicle may bea level 4 vehicle, in which the automated system can control the vehiclein all but a few environments; and/or the autonomous on-road vehicle maybe a level 5 vehicle, in which no human intervention is required and theautomatic system can drive to any location where it is legal to drive.Herein, the terms “autonomous on-road vehicle” and “self-driving on-roadvehicle” are equivalent terms that refer to the same. The term“autonomous on-road vehicle” does not include trains, airplanes, boats,and armored fighting vehicles.

An autonomous on-road vehicle utilizes an autonomous-driving controlsystem to drive the vehicle. The disclosed embodiments may use anysuitable known and/or to be invented autonomous-driving control systems.The following three publications describe various autonomous-drivingcontrol systems that may be utilized with the disclosed embodiments: (i)Paden, Brian, et al. “A Survey of Motion Planning and Control Techniquesfor Self-driving Urban Vehicles.” arXiv preprint arXiv:1604.07446(2016); (ii) Surden, Harry, and Mary-Anne Williams. “TechnologicalOpacity, Predictability, and Self-Driving Cars.” Predictability, andSelf-Driving Cars (Mar. 14, 2016) (2016); and (iii) González, David, etal. “A Review of Motion Planning Techniques for Automated Vehicles.”IEEE Transactions on Intelligent Transportation Systems 17.4 (2016):1135-1145.

Autonomous-driving control systems usually utilize algorithms such asmachine learning, pattern recognition, neural network, machine vision,artificial intelligence, and/or probabilistic logic to calculate on thefly the probability of an imminent collision, or to calculate on the flyvalues that are indicative of the probability of an imminent collision(from which it is possible to estimate the probability of an imminentcollision). The algorithms usually receive as inputs the trajectory ofthe vehicle, measured locations of at least one nearby vehicle,information about the road, and/or information about environmentalconditions. Calculating the probability of an imminent collision is wellknown in the art, also for human driven vehicles, such as theanticipatory collision system disclosed in U.S. Pat. No. 8,041,483 toBreed.

In order to calculate whether a Sudden Decrease in Ride Smoothness(SDRS) event is imminent, the autonomous-driving control system maycompare parameters describing the state of the vehicle at time t₁ withparameters describing the state of the vehicle at time t₂ that isshortly after t₁. If the change in one or more of the parameters reachesa threshold (such as deceleration above a certain value, change ofheight in the road above a certain value, and/or an angular accelerationabove a certain value) then it may be determined that an SDRS event isimminent.

An “occupant” of a vehicle, as the term is used herein refers to aperson that is in the vehicle when it drives. The term “occupant” refersto a typical person having a typical shape, such as a 170 cm tall human(herein “cm” refers to centimeters). An occupant may be a driver, havingsome responsibilities and/or control regarding the driving of thevehicle (e.g., in a vehicle that is not completely autonomous), or maybe a passenger. When an embodiment refers to “the occupant of thevehicle”, it may refer to one of the occupants of the vehicle. Statingthat a vehicle has an “occupant” should not be interpreted that thevehicle necessarily accommodates only one occupant at a time, unlessthat is explicitly stated, such as stating that the vehicle is “designedfor a single occupant”.

Herein, a “seat” may be any structure designed to hold an occupanttravelling in the vehicle (e.g., in a sitting and/or recliningposition). A “front seat” is a seat that positions an occupant it holdsno farther from the front of the vehicle than any other occupants of thevehicle are positioned. Herein, sitting in a seat also refers to sittingon a seat. Sitting in a seat is to be interpreted in this disclosure asoccupying the space corresponding the seat, even if the occupant does soby assuming a posture that does not necessarily correspond to sitting.For example, in some vehicles the occupant may be reclined or lyingdown, and in other vehicles the occupant may be more upright, such aswhen leaning into the seat in a half standing half seating positionsimilar to leaning into a Locus Seat by Focal Upright LLC.

The interchangeable terms “environment outside the vehicle” and “outsideenvironment” refer to the environment outside the vehicle, whichincludes objects that are not inside the vehicle compartment, such asother vehicles, roads, pedestrians, trees, buildings, mountains, thesky, and outer space.

A sensor “mounted to the vehicle” may be connected to any relevant partof the vehicle, whether inside the vehicle, outside the vehicle, to thefront, back, top, bottom, and/or to a side of the vehicle. A sensor, asused herein, may also refer to a camera.

The term “camera” refers herein to an image-capturing device that takesimages of an environment. For example, the camera may be based on atleast one of the following sensors: a CCD sensor, a CMOS sensor, a nearinfrared (NIR) sensor, an infrared sensor (IR), and a device based onactive illumination such as a LiDAR. The term “video” refers to a seriesof images that may be provided in a fixed rate, variable rates, a fixedresolution, and/or dynamic resolutions. The use of a singular “camera”should be interpreted herein as “one or more cameras”. Thus, whenembodiments herein are described as including a camera that capturesvideo and/or images of the outside environment in order to generate arepresentation of the outside environment, the representation may infact be generated based on images and/or video taken using multiplecameras.

Various embodiments described herein involve providing an occupant ofthe vehicle with representation of the outside environment, generated bya computer and/or processor, based on video taken by a camera. In someembodiments, video from a single camera (e.g., which may be positionedon the exterior of the vehicle at eye level), may be sent topresentation to the occupant by the processor and/or computer followinglittle, if any, processing. In other embodiments, video from a singlecamera or multiple cameras is processed in various ways, by the computerand/or processor, in order to generate the representation of the outsideenvironment that is presented to the occupant.

Methods and systems for stitching live video streams from multiplecameras, stitching live video streams with database objects and/or othervideo sources, transforming a video stream or a stitched video streamfrom one point of view to another point of view (such as for generatinga representation of the outside environment for an occupant at eyelevel, or for generating a compartment view for a person standingoutside the compartment), tracking the position of an HMD relative to acompartment, and presenting rendered images that are perfectly alignedwith the outside world—are all known in the art of computer graphics,video stitching, image registration, and real-time 360° imaging systems.The following publications are just a few examples of reviews andreferences that describe various ways to perform the video stitching,image registration, tracking, and transformations, which may be utilizedby the embodiments disclosed herein: (i) Wang, Xiaogang. “Intelligentmulti-camera video surveillance: A review.” Pattern recognition letters34.1 (2013): 3-19. (ii) Szeliski, Richard. “Image alignment andstitching: A tutorial.” Foundations and Trends® in Computer Graphics andVision 2.1 (2006): 1-104. (iii) Tanimoto, Masayuki. “FTV: Free-viewpointtelevision.” Signal Processing: Image Communication 27.6 (2012):555-570. (iv) Ernst, Johannes M., Hans-Ullrich Doehler, and SvenSchmerwitz. “A concept for a virtual flight deck shown on an HMD.” SPIEDefense+ Security. International Society for Optics and Photonics, 2016.(v) Doehler, H-U., Sven Schmerwitz, and Thomas Lueken. “Visual-conformaldisplay format for helicopter guidance.” SPIE Defense+ Security.International Society for Optics and Photonics, 2014. (vi) Sanders-Reed,John N., Ken Bernier, and Jeff GUell “Enhanced and synthetic visionsystem (ESVS) flight demonstration.” SPIE Defense and SecuritySymposium. International Society for Optics and Photonics, 2008. And(vii) Bailey, Randall E., Kevin J. Shelton, and J. J. Arthur III.“Head-worn displays for NextGen.” SPIE Defense, Security, and Sensing.International Society for Optics and Photonics, 2011.

A video that provides “representation of the outside environment” refersto a video that enables the average occupant, who is familiar with theoutside environment, to recognize the location of the vehicle in theoutside environment from watching the video. In one example, the averageoccupant is a healthy 30 years old human who is familiar with theoutside environment, and the threshold for recognizing a video as a“representation of the outside environment” is at least 20 correctrecognitions of the outside environment out of 30 tests.

Herein, sentences such as “VST that represents a view of the outsideenvironment from the point of view of the occupant”, or “VSTrepresentation of the outside environment, which could have been seenfrom the point of view of the occupant” refer to a video representing atleast a portion of the outside environment, with a deviation of lessthan ±20 degrees from the occupant's point of view of the outsideenvironment, and zoom in the range of 30% to 300% (assuming theoccupant's unaided view is at 100% zoom level).

The VST may be generated based on at least one of the followingresources: a video of the outside environment that is taken inreal-time, a video of the outside environment that was taken in the pastand is played/processed according to the trajectory of the vehicle, adatabase of the outside environment that is utilized for rendering theVST according to the trajectory of the vehicle, and/or a video that isrendered as function of locations of physical objects identified in theoutside environment using detection and ranging systems such as RADARand/or LIDAR.

Moreover, the term “video see-through (VST)” covers both directrepresentations of the outside environment, such as a video of theoutside environment, and/or enriched video of the outside environment,such as captured video and/or rendered video of the outside environmentpresented together with one or more layers of virtual objects, as longas more than 20% of the average vehicle occupants, who are familiar withthe outside environment, would be able to determine their location inthe outside environment, while the vehicle travels, without using a map,and with a margin of error below 200 meters. However, it is noted thatshowing a map that indicates the location of the vehicle on the drivingpath (such as from the start to the destination) is not consideredherein as equivalent to the VST, unless the map includes all of thefollowing properties: the map shows images of the path, the images ofthe path capture at least 5 degrees of the occupant's FOV at eye level,and the images of the path reflect the dynamics of the vehicle andchange in a similar manner to a video taken by a camera mounted to thevehicle and directed to the outside environment.

Herein, “field of view (FOV) of the occupant to the outside environment”refers to the part of the outside environment that is visible to theoccupant of a vehicle at a particular position and orientation in space.In one example, in order for an occupant-tracking module to calculatethe FOV to the outside environment of an occupant sitting in a vehiclecompartment, the occupant-tracking module determines the position andorientation of the occupant's head. In another example, in order for anoccupant-tracking module to calculate the FOV of an occupant sitting ina vehicle compartment, the occupant-tracking module utilizes an eyetracker.

It is noted that sentences such as “a three dimensional (3D) videosee-through (VST) that represents a view of the outside environment,which could have been seen from the point of view of the occupant hadthe FOV not been obstructed by at least a portion of the nontransparentelement” cover also just one or more portions of the FOV, and are to beinterpreted as “a three dimensional (3D) video see-through (VST) thatrepresents a view of at least a portion of the outside environment,which could have been seen from the point of view of the occupant had atleast some of the FOV not been obstructed by at least a portion of thenontransparent element”.

The term “display” refers herein to any device that provides a humanuser with visual images (e.g., text, pictures, and/or video). The imagesprovided by the display may be two-dimensional or three-dimensionalimages. Some non-limiting examples of displays that may be used inembodiments described in this disclosure include: (i) screens and/orvideo displays of various devices (e.g., televisions, computer monitors,tablets, smartphones, or smartwatches), (ii) headset- or helmet-mounteddisplays such as augmented-reality systems (e.g., HoloLens),virtual-reality systems (e.g., Oculus rift, HTC Vive, or SamsungGearVR), and mixed-reality systems (e.g., Magic Leap), and (iii) imageprojection systems that project images on a occupant's retina, such as:Virtual Retinal Displays (VRD) that create images by scanning low powerlaser light directly onto the retina, or light-field technologies thattransmit light rays directly into the eye.

Various embodiments may include a reference to elements located at eyelevel. The “eye level” height is determined according to an averageadult occupant for whom the vehicle was designed, who sits straight andlooks to the horizon. Sentences in the form of “an element located ateye level of an occupant who sits in a vehicle” refer to the element andnot to the occupant. The occupant is used in such sentences in thecontext of “eye level”, and thus claims containing such sentences do notrequire the existence of the occupant in order to construct the claim.

Sentences such as “SAEDP located at eye level”, “stiff element locatedat eye level”, and “crumple zone located at eye level” refer to elementsthat are located at eye level, but may also extended to other levels,such as from sternum level to the roof level, from floor level to eyelevel, and/or from floor level to roof level. For example, an SAEDPlocated at an eye level can extend from sternum level to above theoccupant's head, such that at least a portion of the SAEDP is located atthe eye level.

Herein, “normal driving” refers to typical driving conditions, whichpersist most of the time the vehicle is in motion. During normal drivingthe probability of a collision is below a threshold that when reachedtypically involves one or more of the following actions: deployment ofsafety devices that are not usually deployed (e.g., inflating airbags),taking evasive action to avoid a collision, and warning occupants of thevehicle about an imminent event that may cause a Sudden Decrease in RideSmoothness (SDRS).

A Shock-Absorbing Energy Dissipation Padding (SAEDP) is an element thatmay be used to cushion impact of a body during a collision or duringSDRS events. Various types of SAEDPs may be used in embodimentsdescribed herein, such as passive materials, airbags, and pneumaticpads.

Some examples of passive materials that may be used for the SAEDP in oneor more of the disclosed embodiments include one or more of thefollowing materials: CONFOR® foam by Trelleborg Applied Technology,Styrofoam by The Dow Chemical Company, Micro-Lattice Materials and/orMetallic Microlattices (such as by HRL Laboratories in collaborationwith researchers at University of California and Caltech), non-Newtonianenergy Absorbing materials (such as D30® by D30 lab, and DEFLEXION™ byDow Corning®), Sorbothane® by Sorbothane Incorporated, and padding thatincludes compression cells and/or shock absorbers of the Xenith LLC type(such as described in U.S. Pat. No. 8,950,735 and US patent applicationnum. 20100186150), and materials that include rubber such as a spongerubber.

The term “stiff element” refers to a material having stiffness andimpact resistance equal or greater than that of glazing materials foruse in motor vehicles as defined in the following two standards: (i)“American National Standard for Safety Glazing Materials for GlazingMotor Vehicles and Motor Vehicle Equipment Operating on LandHighways-Safety Standard” ANSI/SAE Z26.1-1996, and (ii) The Society ofAutomotive Engineers (SAE) Recommended Practice J673, revised April1993, “Automotive Safety Glasses” (SAE J673, rev. April 93). The term“stiff element” in the context of low speed vehicles refers to amaterial having stiffness and impact resistance equal or greater thanthat of glazing materials for use in low speed motor vehicles as definedin Federal Motor Vehicle Safety Standard 205—Glazing Materials (FMVSS205), from 49 CFR Ch. V (10-1-04 Edition). The stiff element may betransparent (such as automotive laminated glass, or automotive temperedglass) or nontransparent (such as fiber-reinforced polymer, carbon fiberreinforced polymer, steel, or aluminum).

Herein, a nontransparent element is defined as an element having VisibleLight Transmittance (VLT) between 0% and 20%, which does not enable theoccupant to recognize what lies on the other side of it. For example, athick ground glass usually allows light to pass through it but does notlet the occupant recognize the objects on the other side of it, unlikeplain tinted glass that usually lets the occupant recognize the objectson the other side of it, even when it features VLT below 10%. Thenontransparent element includes an opaque element having VLT ofessentially 0% and includes a translucent element having VLT below 20%.VLT is defined as the amount of incident visible light that passesthrough the nontransparent element, where incident light is defined asthe light that strikes the nontransparent element. VLT is also known asLuminous Transmittance of a lens, a light diffuser, or the like, and isdefined herein as the ratio of the total transmitted light to the totalincident light. The common clear vehicle windshield has a VLT ofapproximately 85%, although US Federal Motor Vehicle Safety Standard No.205 allows the VLT to be as low as 70%.

Sentences such as “video unrelated to the VST (VUR)” mean that anaverage occupant would not recognize the video as a representation ofthe outside environment. In some embodiments, the content of the VURdoes not change as function of the position of the occupant's head,which means that the point of view from which the occupant watches theVUR does not change essentially when the occupant's head moves. Herein,stabilization effects, image focusing, dynamic resolution, colorcorrections, and insignificant changes to less than 10% of the frame asfunction of the position of the occupant's head—are still considered ascontent that does not change as function of the position of theoccupant's head. Examples of such content (common in the year 2016)include cinema movies, broadcast TV shows, standard web browsers, andMicrosoft Office 2016 applications (such as Word, Excel and PowerPoint).

Herein, a “crumple zone” refers to a structure designed to slow downinertia and absorb energy from impact during a traffic collision bycontrolled deformation. The controlled deformation absorbs some of theimpact within the outer parts of the vehicle, rather than being directlytransferred to the occupants, while also preventing intrusion intoand/or deformation of the compartment. Crumple zone may be achieved byvarious configurations, such as one or more of the following exemplaryconfigurations: (i) by controlled weakening of sacrificial outer partsof the vehicle, while strengthening and increasing the stiffness of theinner parts of the vehicle, such as by using more reinforcing beamsand/or higher strength steels for the compartment; (ii) by mountingcomposite fiber honeycomb or carbon fiber honeycomb outside thecompartment; (iii) by mounting an energy absorbing foam outside thecompartment; and/or (iv) by mounting an impact attenuator thatdissipates impact.

In order to enable an occupant of a vehicle to view the outsideenvironment, without needing to look out of a physical window, someaspects of this disclosure involve systems that combine videosee-through (VST) with video-unrelated-to-the-VST (VUR).

In one embodiment, a system configured to combine video see-through(VST) with video-unrelated-to-the-VST (VUR) includes a head-mounteddisplay (HMD), a camera, an HMD tracking module and a computer. The HMDis configured to be worn by an occupant of a compartment of a movingvehicle and to present an HMD-video to the occupant. The camera, whichis mounted to the vehicle, is configured to take video of the outsideenvironment (V_(out)). The HMD tracking module is configured tocalculate position of the HMD relative to the compartment, based onmeasurements of a sensor. The computer is configured to receive alocation of a video see-through window (VSTW) in relation to thecompartment, and to calculate, based on the position of the HMD relativeto the compartment, a window-location for the VSTW on the HMD-video.Additionally, the computer is further configured to generate, based onthe window-location and the V_(out), the VST that represents a view ofthe outside environment from the point of view of the occupant. Thecomputer is also configured to generate the HMD-video based on combiningthe VUR with the VST in the window-location. It is to be noted that thecontent of the VUR is unrelated to the video taken by the camera.

In one embodiment, a system configured to combine video see-through(VST) with video-unrelated-to-the-VST (VUR) includes at least thefollowing components: a head-mounted display (HMD), such as HMD 15, acamera (e.g., camera 12), an HMD tracking module 27, and a computer 13.FIG. 1 provides a schematic illustration of at least some of therelationships between the components mentioned above.

The HMD 15 is configured to be worn by an occupant of a compartment of amoving vehicle and to present an HMD-video 16 to the occupant. In oneembodiment, the HMD 15 is an augmented-reality (AR) HMD. In anotherembodiment, the HMD 15 is a virtual reality (VR) HMD. Optionally, inthis embodiment, the system further comprises a video camera mounted tothe VR HMD, and the VST video comprises video of the compartmentreceived from the video camera mounted to the VR HMD. In yet anotherembodiment, the HMD 15 is a mixed reality HMD. The term “Mixed Reality”(MR) as used herein involves a system that is able to combine real worlddata with virtual data. Mixed Reality encompasses Augmented Reality andencompasses Virtual Reality that does not immerse its user 100% of thetime in the virtual world. Examples of mixed reality HMDs include, butare not limited to, Microsoft HoloLens HMD and MagicLeap HMD.

The camera 12, which is mounted to the vehicle, is configured to takevideo of the outside environment (V_(out)). Optionally, the datacaptured by the camera comprises 3D data. For example, the camera may bebased on at least one of the following sensors: a CCD sensor, a CMOSsensor, a near infrared (NIR) sensor, an infrared sensor (IR), and adevice based on active illumination such as a LiDAR.

The HMD tracking module 27 is configured to calculate position of theHMD 15 relative to the compartment, based on measurements of a sensor.In different embodiments, the HMD tracking module 27 may have differentconfigurations.

In one embodiment, the sensor comprises first and second InertialMeasurement Units (IMUs). In this embodiment, the first IMU isphysically coupled to the HMD 15 and is configured to measure a positionof the HMD 15, and the second IMU is physically coupled to thecompartment and is configured to measure a position of the compartment.The HMD tracking module 27 is configured to calculate the position ofthe HMD 15 in relation to the compartment based on the measurements ofthe first and second IMUs.

In another embodiment, the sensor comprises an Inertial Measurement Unit(IMU) and a location measurement system. In this embodiment, the IMU isphysically coupled to the HMD 15 and is configured to measure anorientation of the HMD 15. The location measurement system is physicallycoupled to the compartment and is configured to measure a location ofthe HMD in relation to the compartment. The HMD tracking module 27 isconfigured to calculate the position of the HMD 15 in relation to thecompartment based on the measurements of the IMU and the locationmeasurement system. Optionally, the location measurement system measuresthe location of the HMD 15 in relation to the compartment based on atleast one of the following inputs: a video received from a camera thatcaptures the HMD 15, a video received from a stereo vision system,measurements of magnetic fields inside the compartment, wirelesstriangulation measurements, acoustic positioning measurements, andmeasurements of an indoor positioning systems.

FIG. 2 illustrates one embodiment in which the HMD tracking module 27 isphysically coupled to the compartment and is configured to measure theposition of the HMD relative to the compartment. The HMD tracking module27 may utilize a passive camera system, an active camera system thatcaptures reflections of a projected grid, and/or a real-time locatingsystems based on microwaves and/or radio waves.

The computer 13 is configured to receive a location of a videosee-through window (VSTW) in relation to the compartment, and tocalculate, based on the position of the HMD relative to the compartment,a window-location for the VSTW on the HMD-video. The computer 13 is alsoconfigured to generate, based on the window-location and the V_(out),the VST that represents a view of the outside environment from the pointof view of the occupant. Optionally, the VST is rendered as a 3D videocontent. Additionally, the computer 13 is further configured to generatethe HMD-video 16 based on combining the VUR with the VST in thewindow-location. The computer 13 may use various know in the artcomputer graphics functions and/or libraries to generate the VST,transform the VST to the occupant's point of view, render the 3D videocontent, and/or combine the VUR with the VST.

In one embodiment, the content of the VUR does not change when theoccupant moves the head, and the content of the VUR is unrelated to thevideo taken by the camera. Additionally, the content of the VUR isgenerated based on data that was received more than 2 seconds before theHMD-video 16 is displayed to the occupant. Some examples of the VURinclude a video stream of at least one of the following types ofcontent: a recorded television show, a computer game, an e-mail, and avirtual computer desktop.

FIG. 3 illustrates one embodiment in which the occupant 14 wears an HMD15. The HMD 15 provides video to the occupant 14 through the display ofthe HMD 15. The vehicle includes a camera 12 that takes video of theoutside environment 11 a and processes it in a manner suitable for thelocation of the occupant. The processed video is provided to theoccupant's display in the HMD 15 as a VSTW and the position of the VSTWis calculated in relation to the compartment of the vehicle and moveswith the compartment. While the vehicle is in motion, the VSTW changeits content to represent the outside environment 11 a of the vehicle.Whereas the video-unrelated-to-the-VST doesn't change when the occupantmoves his head. The computer is configured to receive a location of aVSTW in relation to the compartment, and to calculate, based on theposition of the occupant's head, a window-location for the VSTW on thevideo.

FIG. 4 illustrates one embodiment in which the occupant 44 wears HMD 45and views large VUR 40 and smaller VST 41 a. The VUR 40 does not changewhen the occupant's head 44 moves. The VSTW presents video of the streetbased on video taken by the camera that is mounted to the vehicle. Thelocation of the video-see-through window in relation to the compartmentdoes not change when the occupant 44 moves his/her head in order toimitate a physical window that does not change its position relative tothe compartment when the occupant's head moves.

FIG. 5a illustrates how, in one embodiment, the VST moves to the upperleft when the occupant 44 looks to the bottom right. FIG. 5b illustrateshow the VST moves to the bottom right when the occupant 44 looks to theupper left, while the VUR moves with the head. In both cases, the VURmoves with the head while the location of the VST changes according tothe movement of the head relative to the compartment as measured by theHMD tracking module 27.

In some embodiments, the content of the VUR may be augmented-realitycontent, mixed-reality content, and/or virtual-reality content renderedto correspond to the occupant's viewing direction. In this embodiment,the VUR is unrelated to the video taken by the camera. In one example,the VUR may include a video description of a virtual world in which theoccupant may be playing a game (e.g., represented by an avatar).Optionally, in this example, most of the features of the virtual worldare different from the view of the outside of the vehicle (as seen fromthe occupant's viewing direction). For example, the occupant may bedriving in a city, while the virtual world displays woods, a meadow, orouter space. In another example, the VUR may include augmented realitycontent overlaid above a view of the inside of the compartment.

In addition to the components described above, in some embodiments, thesystem may include a second camera that is mounted to the HMD and isconfigured to take video of the compartment (V_(comp)). In thisembodiment, the computer is further configured to generate acompartment-video (CV), based on V_(comp) and a location of acompartment-video window (CVW) in relation to the HMD-video (e.g.,HMD-video 16), and to generate the HMD-video also based on the CV in theCVW, such that the HMD-video combines the VUR with the VST in thewindow-location and with the CV in the CVW. There are various ways inwhich the CVW may be incorporated into the HMD-video. Some examples ofthese approaches are illustrated in the following figures.

FIG. 6 illustrates HMD-video that includes both a non-transparent VST 55in the window-location and a CV 56 that shows the hands of the occupantand the interior of the compartment in the CVW. FIG. 7 illustratesHMD-video that includes both a partially transparent VST 57 in thewindow-location and the CV 56 that shows the hands of the occupant andthe interior of the compartment in the CVW. FIG. 8 illustrates HMD-videothat includes a VST 58 and partially transparent CV 59. The figureillustrates that the occupant sees the outside environment in fullfield-of-view (FOV), while on top of it there is a partially transparentimage (illustrated as dotted image) of the compartment and the hands ofthe occupant, in order to help the occupant not to hit things in thecompartment.

FIG. 9a illustrates HMD-video that includes a VUR 70 in full FOV, afirst window comprising the CV 71 in the CVW and a second smaller windowcomprising the VST 72 in the window-location.

FIG. 9b illustrates HMD-video that includes VUR 70 in full FOV, a firstwindow comprising the CV 71 in the CVW and a second partiallytransparent smaller window comprising the VST 73 in the window-location.

FIG. 10a illustrates HMD-video that includes VUR 70 in full FOV, a firstwindow comprising VST 75 in the window-location and a second smallerwindow comprising zoom out of the CV 76 in the CVW. Optionally, thecabin view in the zoom out is smaller than in reality, and may enablethe occupant to orient in the cabin. Optionally, the occupant may movethe CVW, as illustrated in FIG. 10a where the zoom out of the CV in theCVW is somewhat above its location in reality.

FIG. 10b illustrates HMD-video that includes VUR 70 and a partiallytransparent CV 72. Here a first occupant sees the VUR in fullfield-of-view (FOV), and on top of it there is a partially transparentimage of the compartment and a second occupant that sits to the left ofthe first occupant, which may help the first occupant not to hit thesecond occupant.

There may be various ways in which the system determines the locationand/or size of the VSTW. In one embodiment, the VSTW is pinned to atleast one of the following locations: a specific physical location and alocation of an object in the compartment, such that the location of theVSTW in relation to the compartment does not change when the occupantmoves his/her head with the HMD 15 as part of watching the HMD-video 16and without commanding the VSTW to move in relation to the compartment.

In another embodiment, the system includes a user interface configuredto receive a command from the occupant to move and/or resize the VSTW inrelation to the compartment. In one example, the command is issuedthrough a voice command (e.g., saying “move VST to the bottom”). Inanother example, the command may be issued by making a gesture, which isdetected by a gesture control module in the compartment and/or on adevice of the occupant (e.g., as part of the HMD). Optionally, in thisembodiment, the computer is further configured to: update thewindow-location based on the command from the occupant, and generate anupdated VST based on the updated window-location and the video taken bythe camera. In this embodiment, the VST and the updated VST presentdifferent VSTW locations and/or dimensions in relation to thecompartment. Optionally, the HMD is configured not to present any partof the VST to the occupant when the window-location is not in the fieldof view presented to the occupant through the HMD.

In yet another embodiment, the system may further include a videoanalyzer configured to identify an Object Of Interest (OOI) in theoutside environment. For example, the OOI of interest may be a certainlandmark (e.g., a building), a certain object (e.g., a store or acertain model of automobile), or a person. In this embodiment, thecomputer is further configured to receive, from the video analyzer, anindication of the position of the OOI, and to track the OOI by adjustingthe window-location according to the movements of the vehicle, such thatthe OOI is visible via the VST. Optionally, the HMD is configured not topresent any part of the VST to the occupant when the window-location isnot in the field of view presented to the occupant through the HMD.

The VST that represents the view of the outside environment from thepoint of view of the occupant, in some embodiments, does not necessarilymatch the video taken by the cameras. In one embodiment, the VST mayutilize image enhancement techniques to compensate for outside lightingconditions, to give an occupant an experience similar to looking outthrough a conventional vehicle window but without the view beingdistorted by raindrops or dirt on the window, and/or to improve thevisual impression of the outside environment e.g. by showing backgroundimages which are different from those retrievable from the outsideenvironment. Additionally or alternatively, the VST may mimic theoutside environment, alter the outside environment, and/or be completelydifferent from what can be seen on the outside environment. The VST maybe focused on providing visual information that makes the travellingmore fun. The vehicle may provide different styles of the outsideenvironment to different occupants in the vehicle, such that a first VSTprovided to a first occupant may mimic the outside environment, while asecond VST provided to a second occupant may alter the outsideenvironment and/or be completely different from the outside environment,optionally for comfort enhancement and/or entertainment.

In some embodiments, the VST is informative, and aids at least some ofthe occupants to determine the location of the vehicle in theenvironment. In one embodiment, at least some of those occupants couldnot determine their location without the VST. In one example, less than20% of average vehicle occupants, who are familiar with the outsideenvironment, are able to determine their real location in the outsideenvironment by watching the VUR, without using a map, with a margin oferror that is less than 100 meters, and while the vehicle travels; whilemore than 20% of the average vehicle occupants, who are familiar withthe outside environment, are able to determine their real location inthe outside environment by watching the VST, without using a map, andwith a margin of error that is less than 100 meters, and while thevehicle travels.

FIG. 11a illustrates a FOV in the context of presented video andterminology used herein. The vehicle occupant 200 wears an HMD 201 thatpresents HMD-video (such as HMD-video 16). The HMD-video may bepresented at a single focal plane, or at multiple focal planes,depending on the characteristics of the HMD 201 (when the occupantfocuses on a certain focal plane, then his/her point of gaze is said tobe on the certain focal plane). In addition, the presented objects maybe two-dimensional (2D) virtual objects and/or three-dimensional (3D)virtual objects that may also be referred to as holographic objects.Element 204 represents the location of a nontransparent element fixed tothe vehicle compartment. In one example, the HMD 201 is a holographicHMD, such as Microsoft HoloLens, which can present content displayed ona series of focal planes that are separated by some distance. Thevirtual objects may be presented before the nontransparent element(e.g., polygons 202, 203), essentially on the nontransparent element204, and/or beyond the nontransparent element (e.g., polygons 205, 206).As a result, the occupant's gaze distance may be shorter than thedistance to the nontransparent element (e.g., distance to polygons 202,203), essentially equal to the distance to the nontransparent element204, and/or longer than the distance to the nontransparent element(e.g., distance to polygons 205, 206). Polygon 207 represents a portionof the presented video at eye level of the vehicle occupant, which inone example is within ±7 degrees from the horizontal line of sight.Although the figure illustrates overlapping FOVs of polygons 202, 203,204, and 205, the HMD may show different objects, capturing differentFOVs, at different focal planes. In one example, the HMD may project animage throughout a portion of, or all of, a display volume. Further, asingle object such as a vehicle could occupy multiple volumes of space.

According to the terminology used herein, the nontransparent element 204is said to be located on FOV overlapping the FOV of polygons 205 and 203because polygons 203, 204, 205 share the same FOV. FOV of polygon 206 iscontained in the FOV of polygon 204, and FOV of polygon 207 intersectsthe FOV of polygon 204. FOV of polygon 203 is before the nontransparentelement 204 and therefore may hide the nontransparent element 204partially or entirely, especially when utilizing a multi-focal planeHMD.

FIG. 11b illustrates a FOV in the context of the presented video, wherethe vehicle occupant 210 does not wear an HMD that presents the video,such as when watching an autostereoscopic display. The autostereoscopicdisplay is physically located on plane 214 and the presented video maybe presented at a single focal plane, or at multiple focal planes,depending on the characteristics of the autostereoscopic display. In oneexample, the autostereoscopic display is a holographic display, such asSeeReal Technologies holographic display, where the presented video maypresent virtual objects before the focal plane of the autostereoscopicdisplay (e.g., planes 212, 213), essentially on the focal plane of theautostereoscopic display 214, and/or beyond the focal plane of theautostereoscopic display (e.g., planes 215, 216). As a result, theoccupant's gaze distance may be shorter than the distance to theautostereoscopic display (e.g., planes 212, 213), essentially equal tothe distance to the autostereoscopic display 214, and/or longer than thedistance to the autostereoscopic display (e.g., planes 215, 216). Theterm “autostereoscopic” includes technologies such as automultiscopic,glasses-free 3D, glassesless 3D, parallax barrier, integral photography,lenticular arrays, Compressive Light Field Displays, holographic displaybased on eye tracking, color filter pattern autostereoscopic display,volumetric display that reconstructs light field, integral imaging thatuses a fly's-eye lens array, and/or High-Rank 3D (HR3D).

FIG. 11c illustrates FOV of a 3D camera that is able to capture sharpimages from different focal lengths.

In some embodiments, the vehicle and/or the HMD utilize at least oneInertial Measurement Unit (IMU), and the system utilizes an InertialNavigation System (INS) to compensate imperfections in the IMUmeasurements. An INS typically has one or more secondary navigationsensors that provide direct measurements of the linear velocity,position and/or orientation of the vehicle. These secondary navigationsensors could be anything from stereo vision systems, to GPS receivers,to digital magnetic compasses (DMCs) or any other type of sensor thatcould be used to measure linear velocity, position and/or orientation.In one example, the information from these secondary navigation sensorsis incorporated into the INS using an Extended Kalman Filter (EKF). TheEKF produces corrections that are used to adjust the initial estimationsof linear velocity, position and orientation that are calculated fromthe imperfect IMU measurements. Adding secondary navigation sensors intoan INS can increase its ability to produce accurate estimations of thelinear velocity, position and orientation of the vehicle over longperiods of time.

In one embodiment, the system utilizes domain specific assumptions inorder to reduce drift of an INS used to calculate the HMD spatialposition in relation to the compartment. More specifically, thefollowing methods may be used to reduce or correct drift. Such methodsgenerally fall in the categories of using sensor fusion and/or domainspecific assumptions.

(i) Sensor fusion refers to processes in which signals from two or moretypes of sensors are used to update and/or maintain the state of asystem. In the case of INS, the state generally includes theorientation, velocity and displacement of the device measured in aglobal frame of reference. A sensor fusion algorithm may maintain thisstate using IMU accelerometer and gyroscope signals together withsignals from additional sensors or sensor systems. There are manytechniques to perform sensor fusion, such as Kalman filter and particlefilter.

One example of periodically correcting drift is to use position datafrom a triangulation positioning system relative to the compartment.Such systems try to combine the drift free nature of positions obtainedfrom the triangulation positioning system with the high samplingfrequency of the accelerometers and gyroscopes of the IMU. Roughlyspeaking, the accelerometer and gyroscope signals are used to ‘fill inthe gaps’ between successive updates from the triangulation positioningsystem.

Another example of reducing the drift is using a vector magnetometerthat measures magnetic field strength in a given direction. The IMU maycontain three orthogonal magnetometers in addition to the orthogonalgyroscopes and accelerometers. The magnetometers measure the strengthand direction of the local magnetic field, allowing the north directionto be found.

(ii) In some embodiments, it is possible to make domain specificassumptions about the movements of the occupant and/or the vehicle. Suchassumptions can be used to minimize drift. One example in which domainspecific assumptions may be exploited is the assumption that when thevehicle accelerates or decelerates significantly, the HMD accelerates ordecelerates essentially the same as the vehicle, allowing HMD drift invelocity to be periodically corrected based on a more accurate velocityreceived from the autonomous-driving control system of the vehicle.Another example in which domain specific assumptions may be exploited isthe assumption that when the vehicle accelerates or deceleratessignificantly, the HMDs of two occupants travelling in the same vehicleaccelerate or decelerate essentially the same, allowing HMD drifts to beperiodically corrected based on comparing the readings of the two HMDs.Still another example in which domain specific assumptions are exploitedis the assumption that the possible movement of an HMD of a beltedoccupant is most of the time limited to a portion of the compartment,allowing HMD drifts to be periodically corrected based on identifyingwhen the HMD exceeds beyond that portion of the compartment.

In one example, it may be desirable to adjust the position of displayinga virtual object in response to relative motion between the vehicle andthe HMD so that the virtual object would appear stationary in location.However, the HMD IMU may indicate that the HMD is moving even when thedetected motion is a motion of the vehicle carrying the HMD. In order todistinguish between motion of the HMD caused by the vehicle and motionof the HMD relative to the vehicle, non-HMD sensor data may be obtainedby the HMD from sensor such as an IMU located in the vehicle and/or theGPS system of the vehicle, and the motion of the vehicle may besubtracted from the motion of the HMD in order to obtain arepresentation of the motion of the HMD relative to the vehicle. Bydifferentiating movements of the HMD caused by the occupant motioncompared to movements caused by the vehicle motion, the rendering of thevirtual object may be adjusted for the relative motion between the HMDand the vehicle.

Using the nontransparent element, instead of a transparent glass windowthat provides the same FOV to the outside environment, may providevarious benefits, such as: (i) reduced manufacturing cost of the vehiclecompared to a similar vehicle having instead of the nontransparentelement a transparent glass window that provides the same FOV to theoutside environment as provided by the 3D display device, (ii) reducedweight of the vehicle compared to a similar vehicle having instead ofthe nontransparent element a transparent glass window that provides thesame FOV to the outside environment as provided by the 3D displaydevice, and provides the same safety level, (iii) better aerodynamicshape and lower drag for the vehicle, which results in an improvedenergy consumption, and (iv) improved privacy for the occupant as aresult of not enabling an unauthorized person standing nearby thevehicle to see the occupant directly.

The term “real-depth VST window (VSTW)” is defined herein as an imagingdisplay that shows a 3D image of an outside environment located beyond awall that interrupts the occupant's unaided view of the outsideenvironment. The real-depth VSTW has the following characteristics: (i)the 3D image corresponds to a FOV to the outside environment beyond thewall, as would have essentially been seen by the occupant had the wallbeen removed; (ii) the outside environment is captured by a camera, andthe rendering of the 3D image is based on images taken by the camera;and (iii) while looking via the imaging display, the occupant's point ofgaze (where one is looking) is most of the time beyond the wall thatinterrupts the occupant's unaided view of the outside environment.

A possible test to determine whether “(i) the 3D image corresponds to aFOV to the outside environment beyond the wall, as would haveessentially been seen by the occupant had the wall been removed” iswhether an imaginary user standing beyond the wall, watching both thereal-depth VSTW and the outside environment, would recognize that atleast 20% of the contours of objects in the 3D image correspond to thecontours of the objects seen on the outside environment. Differencesbetween the colors of the corresponding objects in the 3D image and theoutside environment usually do not affect the criterion of the 20%corresponding contours, as long as the color difference does not affectthe perception of the type of object. For example, different skin colorsto corresponding people in the 3D image and the outside environment donot violate the criterion of the 20% corresponding contours. As anotherexample, differences in the weight and/or height of correspondingobjects in the 3D image and the outside environment do not violate thecriterion of the 20% corresponding contours as long as the imaginaryuser understands that the objects correspond to the same person.

Sentences such as “from the FOV of the occupant” are to be interpretedas no more than 20 degrees angular deviation from the field of view ofthe occupant to the outside environment. Zoom in/out does not affect theFOV as long as the average occupant would still recognize the renderedenvironment as the 3D VST. For example, zoom in of up to ×4, whichmaintains no more than 20 degrees angular deviation from the FOV of theoccupant to the outside environment, is still considered “from the FOVof the occupant”. Reasonable lateral deviation essentially does notaffect the FOV as long as the average occupant would still recognize therendered environment as the 3D VST. For example, displaying to theoccupant the outside environment from the FOV of a camera located on theroof of the occupant's vehicle, is still considered as showing theoutside environment from the occupant's FOV.

A possible test to determine whether “(ii) the outside environment ismeasured by a camera, and the images taken by the camera are used torender the 3D image” is whether the real-depth VSTW would display adifferent 3D VST when it does not receive the images taken by thecamera. For example, assuming the camera is a 3D video camera, and the3D image is a manipulation of the images taken by the 3D video camera;then, when the real-depth VSTW does not receive the images, it cannotshow the changes that are taking place in the outside environment. Asanother example, assuming the 3D image is mainly rendered from cacheddata stored in a database, and the camera is used to provide the setupof objects that behave in an unknown way, such as trajectories of nearbyvehicles on the road, or a gesture of a person walking beyond the wall;then, when the output of the camera is used to render the 3D image, thereal-depth VSTW would represent the unknown trajectory of the nearbyvehicles or the unknown gesture of the person, while when the output ofthe camera is not used to render the 3D image, the real-depth VSTW wouldnot represent the unknown trajectory of the nearby vehicles or theunknown gesture of the person merely because the renderer does not havethat data.

A possible test to determine whether “(iii) the occupant's point of gaze(where one is looking) is most of the time beyond the wall thatinterrupts the occupant's unaided view of the outside environment”includes the following steps: (a) use eye tracker to determine the pointof gaze on a representative scenario, (b) measure the distance to thewall, and (c) determine whether the average gaze distance is longer thanthe distance to the wall.

Some aspects of this disclosure involve a system that utilizes windowshading of a vehicle window in order to improve the quality of videoviewed by an occupant of the vehicle who wears a head-mounted display(HMD).

In one embodiment, an autonomous on-road vehicle includes a systemconfigured to enable an HMD to cooperate with a window light shadingmodule. This embodiment involves a light shading module, a camera, aprocessor, and the HMD. The light shading module is integrated with avehicle window and is configured to be in at least one of first andsecond states. In the first state the Visible Light Transmittance (VLT)of the vehicle window is above 10% of ambient light entering through thewindow, in the second state the VLT of the vehicle window is below 50%of ambient light entering through the window, and the VLT of the vehiclewindow in the first state is higher than the VLT of the vehicle windowin the second state. The camera is fixed to the vehicle and configuredto take video of the outside environment. The processor is configured togenerate, based on the video, a video see-through (VST) that representsthe outside environment from a point of view of an occupant looking tothe outside environment through at least a portion of the vehiclewindow. The HMD comprises an optical see-through component and a displaycomponent; the HMD is configured to operate according to a first mode ofoperation when the occupant looks at the direction of the vehicle windowand the light shading module is in the first state, and to operateaccording to a second mode of operation when the occupant looks at thedirection of the vehicle window and the light shading module is in thesecond state. Wherein the total intensity of the VST light, emitted bythe display component and reaching the occupant's eyes, is higher in thesecond mode than in the first mode.

In one embodiment, a system configured to enable a head-mounted display(HMD) to cooperate with a window light shading module of an autonomouson-road vehicle includes at least the following elements: the HMD 62, alight shading module 61, a camera (such as camera 12), and a processor18. FIG. 12 is a schematic illustration of at least some of therelationships between the system elements mentioned above.

The light shading module 61 is integrated with a vehicle window and isconfigured to be in at least one of first and second states. Optionally,the light shading module 61 covers more than half of the frontwindshield in the second state. In one embodiment, in the first state,the Visible Light Transmittance (VLT) of the vehicle window is above 10%of ambient light entering through the window, and in the second state,the VLT of the vehicle window is below 50% of ambient light enteringthrough the window. Additionally, the VLT of the vehicle window in thefirst state is higher than the VLT of the vehicle window in the secondstate. In another embodiment, in the first state the VLT of the vehiclewindow is above 70% of ambient light entering through the window, and inthe second state, the VLT of the vehicle window is below 30% of ambientlight entering through the window.

Herein, “ambient light” in the context of a vehicle refers to visiblelight that is not controlled by the vehicle, such as light arrivingfrom: the sun, lights of other vehicles, street/road lighting, andvarious reflections from elements such as windows.

In some embodiments, utilizing the light shading module 61 may improvethe quality of images viewed via the HMD 62 when the light shadingmodule 61 is in the second state. Optionally, the perceived contrast ofthe optical see-through component is better when the light shadingmodule is in the second state compared to when the light shading module61 is in the first state.

Various types of light shading modules may be utilized in embodimentsdescribed herein. In one embodiment, the light shading module 61 is amovable physical element configured to reduce the intensity of theambient light entering into the vehicle compartment through the vehiclewindow. Optionally, the light shading module is unfurled on the insideof the compartment in order to block at least 50% of the ambient lightintensity. Optionally, the light shading module is unfurled on theoutside of the compartment in order to block at least 50% of the ambientlight intensity. FIG. 13a illustrates a first mode where the occupantsees the outside environment through the optical see-through component.This figure illustrates the view that the occupant sees when lookingoutside through the window. FIG. 13b illustrates a second mode where theoccupant sees the outside environment through the VST. In this example,the outside environment is a bit different, and there is also a virtualSuperman floating near the tree.

In another embodiment, the light shading module 61 may be a curtain.FIG. 14 illustrates a VST over a curtain. FIG. 15 illustrates a lightshading module that is unfurled on the inside of the compartment. FIG.16 illustrates a light shading module that is unfurled on the outside ofthe compartment.

And in yet another embodiment, the vehicle window is made of a materialthat is able to serve as the light shading module 61 by changing itstransparency properties.

The camera is fixed to the vehicle, and configured to take video of theoutside environment. For example, the camera may be based on at leastone of the following sensors: a CCD sensor, a CMOS sensor, a nearinfrared (NIR) sensor, an infrared sensor (IR), and a device based onactive illumination such as a LiDAR.

The processor is configured to generate, based on the video, a videosee-through (VST 19) that represents the outside environment from apoint of view of an occupant looking to the outside environment throughat least a portion of the vehicle window. Optionally, the processor isfurther configured not to generate the VST 19 when the HMD 62 operatesin the first mode.

The HMD 62 comprises an optical see-through component and a displaycomponent. Optionally, the HMD 62 is configured to operate according toa first mode of operation when the occupant looks at the direction ofthe vehicle window and the light shading module 61 is in the firststate, and to operate according to a second mode of operation when theoccupant looks at the direction of the vehicle window and the lightshading module 61 is in the second state. The total intensity of the VSTlight, emitted by the display component and reaching the occupant'seyes, is higher in the second mode than in the first mode.

In one embodiment, in the first mode, intensity of light that reachesthe occupant's eyes via the optical see-through component is higher thanintensity of light from the VST that is emitted by the display componentand reaches the occupant's eyes. And in the second mode, the intensityof light from the environment that reaches the occupant's eyes via theoptical see-through component is lower than the intensity of light fromthe VST that is emitted by the display component and reaches theoccupant's eyes. In one example, the total intensity of VST light,emitted by the display component and reaching the occupant's eyes, is atleast 50% higher in the second mode than in the first mode. In someembodiments, the display component may be based on a digital displaythat produces the virtual image (such as in Oculus rift), direct retinaillumination (such as in Magic Leap), or other methods that are capableof producing the virtual image.

In one embodiment, the system described above optionally includes anoccupant tracking module configured to calculate the point of view ofthe occupant based on measurements of a sensor. Optionally, the occupanttracking module is the HMD tracking module 27. Optionally, in thisembodiment, the processor is further configured to render the VST basedon data received from the occupant tracking module. Optionally, thedisplay is a three dimensional (3D) display configured to show theoccupant the VST, such that point of gaze of the occupant, while lookingvia the 3D display device, is most of the time beyond the location ofthe light shading module 61.

In some embodiments, an occupant of a vehicle may have the opportunityto view video see-through (VST), which is video generated based on videoof the environment outside the vehicle. VST can often replace the needto look out of a window (if the vehicle has windows). Some examples ofscenarios in which VST may be available in a vehicle include awindowless vehicle, a vehicle with shaded windows having VLT below 30%,and/or when the occupant wears a VR headset. While traveling in such avehicle, the occupant may benefit from gaining a view to the outsideenvironment when an unexpected driving event occurs. By being made awareof the event, the occupant is less likely to be surprised, disturbed,and/or distressed by the event.

While traveling in a vehicle, an occupant of the vehicle may not alwaysbe aware of the environment outside and/or of what actions the vehicleis about to take (e.g., breaking, turning, or hitting a speedbump).Thus, if such an event occurs without the occupant being aware that itis about to happen, this may cause the occupant to be surprised,disturbed, distressed, and even physically thrown off balance (in a casewhere the event involves a significant change in the balance of thephysical forces on the occupant). This type of event is typicallyreferred to herein as a Sudden Decrease in Ride Smoothness (SDRS) event.Some examples of SDRS events include at least one of the followingevents: hitting a speed bump, driving over a pothole, climbing on thecurb, making a sharp turn, a hard breaking, an unusual acceleration(e.g., 0-100 km/h in less than 6 seconds), and starting to drive after afull stop.

In some embodiments, an SDRS event takes place at least 2 minutes afterstarting to travel and it is not directly related to the act of thestarting to travel. Additionally, the SDRS event takes place at least 2minutes before arriving to the destination and is not directly relatedto the act of arriving at the destination. In one example, a sentencesuch as “an SDRS event is imminent” refers to an SDRS event that is: (i)related to traveling in the vehicle, and (ii) expected to happen in lessthan 30 seconds, less than 20 seconds, less than 10 seconds, or lessthan 5 seconds. In another example, a sentence such as “an SDRS event isimminent” may refer to an event that starts at that instant, or is aboutto start within less than one second.

The following is a description of an embodiment of a video system thatmay be used to increase awareness of an occupant of a vehicle regardingan imminent SDRS. In one embodiment, a video system for an autonomouson-road vehicle includes at least autonomous-driving control system 65,a camera (such as camera 12), a processor (such as processor 18), andvideo module 66. FIG. 17 is a schematic illustration of at least some ofthe relationships between the system elements mentioned above.

The autonomous-driving control system 65 is configured to generate,based on trajectory of the vehicle and information about the road, anindication indicative of whether a Sudden Decrease in Ride Smoothness(SDRS) event is imminent. Optionally, the autonomous-driving controlsystem 65 receives at least some of the information about the road fromat least one of the following sources: sensors mounted to the vehicle,sensors mounted to nearby vehicles, an autonomous-driving control system65 used to drive a nearby vehicle, and a database comprisingdescriptions of obstacles in the road that are expected to cause intensemovement of the vehicle. In one example, the database comprising thedescriptions of the obstacles includes one or more of the followingtypes of data: locations of speed bumps, locations of potholes,locations of stop signs, and locations of sharp turns in the road.

In one embodiment, the autonomous-driving control system 65 isconfigured to generate the indication indicative of whether an SDRSevent is imminent based on at least one of the following configurations:(i) the autonomous-driving control system 65 receives images of the roadfrom a camera, and calculates the indication based on the vehicletrajectory and image analysis of the images, (ii) the autonomous-drivingcontrol system 65 receives from a radar reflections of electromagneticwaves from the road, and calculates the indication based on the vehicletrajectory and signal processing of the reflections, and (iii) theautonomous-driving control system 65 receives a notification from adetailed road map, and calculates the indication based on the vehicletrajectory and the notification.

The camera, which is mounted to the vehicle, is configured to take videoof the environment outside the vehicle. Optionally, the data captured bythe camera comprises 3D data. The processor is configured to generate avideo see-through (VST) based on the video taken by the camera.

The video module 66 is configured to select a first mode ofpresentation, in which a video-unrelated-to-the-VST (VUR) is presentedon the foveal vision region of the occupant, at eye level, responsive tothe indication not indicating that an SDRS event is imminent. The videomodule 66 is further configured to select a second mode of presentation,in which the VST is presented on the foveal vision region of theoccupant, at eye level, responsive to the indication indicating that anSDRS event is imminent. Optionally, the VST captures more than 50% ofthe foveal vision region of the occupant in the second mode ofpresentation. In some embodiments, presenting video on the foveal visionregion comprises presenting images with at least 50% transparency.Herein, “foveal vision” refers to an angle of about 5° of the sharpestfield of vision.

In one embodiment, in the first mode of presentation, the VUR ispresented on the foveal vision region of the occupant with opacity A,and the VST is presented on the foveal vision region of the occupantwith opacity B, where A>B>0. Optionally, a normalized opacity parametertakes a value from 0.0 to 1.0, and the lower the value the moretransparent the video is. In this embodiment, in the second mode ofpresentation, the VUR is presented on the foveal vision region of theoccupant with opacity A′, and the VST is presented on the foveal visionregion of the occupant with opacity B′, where B′>B and B′>A′. Inoptional embodiments, one or more of the following values may be true:A′>0, B=0, and A′=0. Herein, “partially transparent” refers to opacitybelow one and above zero.

Having the VST presented when an SDRS event is imminent can make theoccupant be aware and prepared for the SDRS event. Thus, the occupant isless likely to be startled, distressed, and/or physically thrown offbalance by the SDRS event. In one example, the SDRS event involveshitting a speedbump, while the occupant views a movie. About 5 secondsprior to hitting the speedbump, a partially transparent windowdisplaying VST in which the speedbump is highlighted (e.g., flashingred) is presented on the foveal vision region of the occupant for acouple seconds (e.g., by being presented in the center of the movie).This way upon hitting the speedbump, the occupant is not startled by theevent. In another example, the autonomous-driving control system 65determines that a “hard breaking” is required, e.g., in order to avoidcollision with a vehicle ahead that slowed unexpectedly. In thisexample, the occupant may be working on a virtual desktop, and within100 milliseconds of when the determination is made that the vehicle isabout to rapidly deaccelerate (a “hard breaking”), the VST depicting therear of the vehicle ahead is displayed in the center of the virtualdesktop. This way the occupant is immediately made aware of why thevehicle is breaking and this notification may prompt the occupant toseek a more appropriate posture for the breaking.

Some illustrations of utilization of the different modes of operationare given in the following figures. FIG. 18a illustrates presenting VURresponsive to not receiving from the autonomous-driving control system65 an indication that an SDRS event is imminent. This figure has twoparts, the left part shows the vehicle driving over a clean road, andthe right part shows the VUR. FIG. 18b illustrates presenting VSTresponsive to receiving from the autonomous-driving control system 65 anindication that an SDRS event is imminent. The figure has two parts, theleft part shows the vehicle about to drive over a pothole, and the rightpart shows a small window showing the pothole over the VUR (optionallyto warn the occupant). FIG. 18c illustrates presenting VST responsive toreceiving from the autonomous-driving control system 65 an indicationthat an SDRS event is imminent. The figure has two parts, the left partshows the vehicle about to enter a sharp turn, and on the right partshows a small window showing the sharp turn over the VUR (optionally towarn the occupant).

Traditional vehicles typically have a front windshield that offersoccupants of the vehicle a frontal view of the outside environment.However, in some embodiments, this frontal view may be provided usingthe VST. For example, in one embodiment, the vehicle includes anontransparent element, which is coupled to the vehicle, and obstructsat least 30 degrees out of the frontal horizontal unaided FOV to theoutside environment of an occupant at eye level. In one example of astandard vehicle, such as Toyota Camry model 2015, the frontalhorizontal unaided FOV extends from the left door through the windshieldto the right door.

The use of the nontransparent element improves the safety of theoccupant during a collision compared to a similar vehicle having thesame total weight and comprising a transparent glass window instead ofthe nontransparent element. The nontransparent element may be coupled tothe vehicle in various configurations, in embodiments described herein.In one embodiment, the nontransparent element is fixed to the vehicle atan angle, relative to the occupant, that is covered by the field of viewof the VST, and the nontransparent element features visible lighttransmittance (VLT) below 10% of ambient light.

Various types of displays may be utilized to present the occupant withvideo (e.g., the VST and/or the VUR). In one embodiment, the video ispresented to the occupant on a screen coupled to the vehiclecompartment. In one example, the screen coupled to the vehiclecompartment utilizes parallax barrier technology. A parallax barrier isa device located in front of an image source, such as a liquid crystaldisplay, to allow it to show a stereoscopic image or multiscopic imagewithout the need for the viewer to wear 3D glasses. The parallax barrierincludes a layer of material with a series of precision slits, allowingeach eye to see a different set of pixels, thus creating a sense ofdepth through parallax. In another embodiment, the occupant wears ahead-mounted display (HMD), and the HMD is used to present the video tothe occupant. Optionally, the HMD is a VR headset, and as a result ofpresenting the VST, the occupant does not need to remove the VR headsetin order to see the cause of the SDRS event.

In some embodiments, the video module 66 may be selective regardingindications of which SRDS events may prompt it to operate in the secondmode of operation. For example, if the occupant is engaged in a game,the video module 66 may refrain from presenting the VST in the fovealvision region if the vehicle is about to make a sharp turn. However, itmay optionally still present the VST in the foveal vision region if theSDRS event involves something that may be more forcefully felt by theoccupant, such as extreme evasive maneuvering performed to avoid acollision.

In some embodiments, the video module 66 may determine whether to show aVST responsive to an SRDS event (in the second mode of operation) basedon the level of concentration of the occupant. For example, if theoccupant is deeply engaged in a certain activity (e.g., in work orplaying a game) above a threshold, the video module 66 may refrain fromoperating in the second mode for certain SDRS events that would causethe video module 66 to operate in the second mode were the occupantengaged in the certain activity below the threshold. In one example, theengagement level may be based on the occupant's level of concentration,as measured by a wearable sensor (such as an EEG headset or asmartwatch) or a sensor fixed to the compartment (such as an eyetracker, a thermal camera, or a movement sensor embedded in the seat).

Presenting an occupant of a vehicle with video see-through (VST) of theoutside environment from a point of view of the occupant can help theoccupant be prepared for various events that may be considered to causea Sudden Decrease in Ride Smoothness (SDRS events). Some examples ofSDRS events include the following events: hitting a speed bump, drivingover a pothole, climbing on the curb, making a sharp turn, a hardbreaking, an unusual acceleration (e.g., 0-100 km/h in less than 6seconds), and starting to drive after a full stop.

In order for the occupant to become aware of an imminent SDRS event, theVST needs to be presented in an attention-grabbing way. For example,when an SDRS event is imminent, the VST that describes the environmentis brought to the center of the occupant's attention by displaying it ateye level and/or increasing the size of the VST (compared to other timeswhen an SDRS event is not imminent).

The following is a description of an embodiment of a video system thatmay be used to increase awareness of an occupant of a vehicle regardingan imminent SDRS by making the VST more prominent for an imminent SDRS.In one embodiment, a video system for an autonomous on-road vehicleincludes at least the autonomous-driving control system 65, a camera,and a processor. In this embodiment, the occupant is engaged, at leastpart of the time, in entertainment- or work-related activities, whichinvolve presentation of video-unrelated-to-the-VST (VUR) to theoccupant, for example, on a screen coupled to the compartment of thevehicle or a HMD worn by the occupant. Some examples of such content(common in the year 2016) include cinema movies, broadcast TV shows,standard web browsers, and Microsoft Office 2016 applications (such asWord, Excel and PowerPoint).

The camera, which is mounted to the vehicle, is configured to take videoof the environment outside the vehicle. The processor is configured togenerate, based on the video taken by the camera, a video see-through(VST) of outside environment from a point of view of an occupant of thevehicle. Optionally, the occupant is in a front seat of the vehicle(such that no other occupant in the vehicle is positioned ahead of theoccupant). In some embodiments, the processor is configured to presentvideo, which may include VUR and/or VST, to the occupant using differentpresentation modes, depending on whether an SDRS event is imminent. Forexample, the video may be presented according to first or second modes,depending on whether an SDRS event is imminent. Optionally, the VSTcaptures in the video according to the first mode a diagonal FOV of atleast 3°, 5°, or 10° of the occupant's FOV. Optionally, the VST is notpresented in the foveal vision region of the occupant in the videoaccording to the first mode, while the VST is presented in the fovealvision region of the occupant in the video according to the second mode.

In one embodiment, responsive to an indication that is not indicative ofan imminent SDRS event (generated by the autonomous-driving controlsystem 65), the processor is configured to provide the occupant videoaccording to the first mode. In the video according to the first mode,the occupant is presented with video that comprises avideo-unrelated-to-the-VST (VUR) at eye level in the direction offorward traveling. Additionally, the video may comprise a videosee-through (VST) of outside environment that is not presented at eyelevel in the direction of forward traveling.

Receiving an indication indicative that an SDRS event is imminent maychange the way video is presented to the occupant. Optionally, thischange is made without receiving a command to do so from the occupant.In one embodiment, responsive to the indication indicating that an SDRSevent is imminent, the processor is configured to provide video to theoccupant video according to the second mode. In the video according tothe second mode, the occupant is presented with video that comprises theVST, presented at eye level in the direction of forward traveling.Optionally, if the video according to the first mode includes VST, thenthe size of the VST window in the video according to the second mode islarger by at least 25% relative to the size of the VST window in thevideo according to the first mode. Optionally, the video according tothe second mode includes presenting the VUR in the background (e.g., theVST is overlaid above the VUR). Optionally, while providing the videoaccording to the second mode, responsive to an updated indication thatdoes not indicate that an SDRS event is imminent, the processor isfurther configured to switch back to provide the video according to thefirst mode to the occupant.

The following figures illustrate various ways in which the videosaccording to the first and second modes may be utilized. FIG. 19aillustrates presenting a VUR, which is a movie showing a person skiing,responsive to not receiving from the autonomous-driving control system65 an indication that an SDRS event is imminent. This figure has twoparts, the left part shows the vehicle driving over a clean road, andthe right part shows the VUR with a small VST on the right. FIG. 19billustrates presenting a VST responsive to receiving from theautonomous-driving control system 65 an indication that an SDRS event isimminent. This figure has two parts, the left part shows the vehicleabout to drive over a speed bump, and the right part shows the VUR butnow with a big VST on the right. In this example, the big VST capturesabout half of the VUR and shows the speed bump. FIG. 19c illustratespresenting a partially transparent VST responsive to receiving from theautonomous-driving control system 65 an indication that an SDRS event isimminent. Here, the big VST (that captures about half of the VUR andshows the speed bump) is presented as partially transparent layer overthe VUR in order to show the occupant both the VUR and the VST.

In one embodiment, presenting to the occupant video according to thesecond mode involves presenting the VUR behind the VST, and the size andlocation of the VUR in the video according to the second mode isessentially the same as the size and location of the VUR in the videoaccording to the first mode. Optionally, this means that there is adifference of less than 10% in the size and location of the VURs in thevideos according to the first and second modes. In another embodiment,the VUR is presented in a diagonal FOV of at least 10 degrees, and isnot based on the video taken by the camera.

In some embodiments, the VUR may be unrelated to the purpose of thetraveling in the vehicle. For example, the VUR may include videosrelated to the following activities: watching cinema movies, watching TVshows, checking personal emails, playing entertainment games, andsurfing in social networks.

In some embodiments, the occupant's field of view (FOV) to the outsideenvironment is obstructed by a nontransparent element, and the VSTrepresents at least a portion of the obstructed FOV. Optionally, theoccupant uses a VR headset and the obstruction is due to anontransparent element belonging to the VR headset. Additionally oralternatively, the obstruction may be due to the vehicle's compartment;in this case the nontransparent element may be an SAEDP, a safety beam,and/or a crumple zone at eye level, which obstruct at least 30 degreesout of the frontal horizontal unaided FOV to the outside environment ofthe occupant at eye level.

When traveling in a vehicle, an occupant of the vehicle may not alwaysbe viewing the outside environment. For example, the occupant may beengaged in work- or entertainment-related activities. Additionally, insome vehicles, the occupant may not have a good view of the outsideenvironment most of the time, or even all of the time. For example, thevehicle may have very few (or no) windows, or the vehicle may have ashading mechanism that reduces the light from the outside. However,there are times when the occupant should be made aware of the outsideenvironment, even though the occupant may not be actively driving thevehicle. For example, the occupant may be made aware of the outsideenvironment in order to make the occupant prepared for an event thatcauses a Sudden Decrease in Ride Smoothness (an SDRS event). Someexamples of SDRS events include the following events: hitting a speedbump, driving over a pothole, climbing on the curb, making a sharp turn,a hard breaking, an unusual acceleration (e.g., 0-100 km/h in less than6 seconds), and starting to drive after a full stop.

In order for the occupant to become aware of an imminent SDRS event, insome embodiments that involve a vehicle that has a shading module thatcontrols how much ambient light is let in, when an SDRS event isimminent the vehicle may increase the amount of light that enters via awindow. This additional light can give an occupant a better view of theoutside environment, which can make the occupant aware and betterprepared for the SDRS.

The following is a description of an embodiment of a system that may beused to increase awareness of an occupant of a vehicle regarding animminent SDRS by enabling more ambient light to enter a vehicle via awindow. In one embodiment, a shading system for a window of anautonomous on-road vehicle includes at least the autonomous-drivingcontrol system 65, a shading module, and a processor.

FIG. 20a illustrates a smart glass shading module that operatesaccording to an indication that an SDRS event is not imminent. Thisfigure has two parts, the left part shows the vehicle driving over aclean road, and the right part shows that the smart glass window blocksmost of the ambient light (illustrated in the figure by the tree outsidethat is invisible to the occupant). FIG. 20b illustrates the smart glassshading module that operates according to an indication that an SDRSevent is imminent. This figure has two parts, the left part shows thevehicle about to drive over a pothole, and the right part shows that thesmart glass window does not block the ambient light (illustrated in thefigure by the tree outside that is visible to the occupant).

The shading module is configured to control the amount of ambient lightthat enters the vehicle via the window. Optionally, the window is afront-facing window (e.g., a windshield). Optionally, the window is aside-facing window. There are various types of shading modules that maybe utilized in different embodiments.

In one embodiment, the shading module comprises a curtain. Optionally,the curtain covers most of the area of the window. Optionally, thecurtain may open and close with the aid of an electromechanical device,such as a motor, based on commands issued by the processor.

In another embodiment, the shading module is a movable physical elementconfigured to reduce the intensity of the ambient light entering throughthe vehicle window into the vehicle compartment. For example, theshading module may include various forms of blinds, a shutter, or asliding element. Optionally, the shading module may be unfurled on theinside of the vehicle compartment in order to block more than 70% of theambient light intensity. Optionally, the shading module may be unfurledon the outside of the vehicle compartment in order to block more than70% of the ambient light intensity.

In yet another embodiment, the shading module comprises a smart glassable to change its light transmission level. Optionally, the smart glassis a vehicle window smart glass that comprises suspended particledevices (SPDs) film. Smart glass window may also be known as aswitchable glass, a smart window, and/or a switchable window. Smartglass is glass or glazing whose light transmission properties arealtered when voltage, current, light or heat is applied. Examples ofelectrically switchable smart glass include: suspended particle devices(SPDs), electrochromic devices, transition-metal hydride electrochromicsdevices, modified porous nano-crystalline films, polymer dispersedliquid crystal devices, micro-blinds, and thin coating of nanocrystalsembedded in glass. Examples of non-electrical smart glass include:mechanical smart windows, Vistamatic®, and Sunvalve.

The processor is configured to command the shading module to operate indifferent modes based on indications generated by the autonomous-drivingcontrol system 65. In some embodiments, the processor is configured tocommand the shading module to operate in different modes that allowdifferent amounts of the ambient light to enter the vehicle via thewindow, depending on whether an SDRS event is imminent. For example,shading module may operate in first or second modes, depending onwhether an SDRS event is imminent. Optionally, in the first mode theshading module blocks more of the ambient light entering through thevehicle window than in the second mode. Optionally, the increasedambient light in the second mode can help make the occupant more awareof the outside environment, which can enable the occupant to prepare forthe SDRS event.

In one embodiment, responsive to an indication that no SDRS event isimminent, the processor is configured to command a shading module tooperate in a first mode in which the shading module blocks more than 30%of ambient light entering through a window of the vehicle. Receiving anindication indicative that an SDRS event is imminent may change theamount of ambient light that enters the vehicle via the window.Optionally, this change is made without receiving a command to do sofrom the occupant. In one embodiment, responsive to an indication thatan SDRS event is imminent, the processor is configured to command theshading module to operate in the second mode in which the shading moduleblocks less than 90% of the ambient light entering through the vehiclewindow.

Some aspects of this disclosure include a system that can show anoccupant of a vehicle and Object Of Interest (OOI) in the outsideenvironment, which the occupant would otherwise miss (e.g., due to beingengaged in work- or entertainment-related activities and/or having nodirect view of the outside environment).

In one embodiment, a system configured to identify an Object Of Interest(OOI) in the outside environment, and to present to an occupant of anautonomous on-road vehicle a video see-through (VST) that comprises theOOI, includes at least a camera (such as camera 12), a processor (suchas processor 18), and a video module (such as video module 66).

Different types of things may be considered an OOI in differentembodiments. In one embodiment, the OOI is selected from a setcomprising: types of vehicles, types of scenery, and types of people.Optionally, the system may include a user interface configured topresent a menu that enables the occupant to select the types of OOI,which when identified, will be presented in the second mode ofpresentation. In one example, an OOI may be any type of high-end vehicle(e.g., a Porsche). In another example, an OOI may include an ocean view.

The camera, which is mounted to the vehicle, is configured to take videoof the environment outside the vehicle. Optionally, the data captured bythe camera comprises 3D data. The processor is configured to generate,based on the video taken by the camera, a video see-through (VST) ofoutside environment from a point of view of an occupant of the vehicle.Optionally, the occupant is in a front seat of the vehicle (such that noother occupant in the vehicle is positioned ahead of the occupant).

The video module is configured to present, to the occupant, video thatcaptures diagonal field of view (FOV) of at least 10 degrees, at eyelevel. Optionally, the video module is further configured to select amode of operation based on whether an indication is received that isindicative of whether video taken by the camera include the OOI.Optionally, changing a mode of operation is done without an expressedcommand at that time by the occupant.

In one embodiment, when not receiving an indication that the imagesincludes the OOI, the video module is configured to select a first modeof presentation, in which a video-unrelated-to-the-VST (VUR) ispresented on the foveal vision region of the occupant. And responsive toreceiving the indication that the images include the OOI, the videomodule is further configured to select a second mode of presentation, inwhich the VST is presented on the foveal vision region of the occupant.

There are various ways in which an indication of the OOI may begenerated in different embodiments. In one embodiment, the systemincludes an image processing module configured to identify the OOI inthe VST and to generate the indication. Various image processingtechniques and/or image identification methods known in the art may beutilized by the image processing module for this task.

In another embodiment, the system may include the autonomous-drivingcontrol system 65 configured to utilize a positioning system, such asGPS coordinates, to identify that the vehicle reached the OOI and togenerate the indication. Optionally, the occupant may define whichplaces may be OOIs and/or what types of locations are to be consideredOOIs (e.g., historical monuments, nice scenery, etc.).

In still another embodiment, the system may include a crowd-based moduleconfigured to identify the OOI based on feedback received from manydifferent occupants who watched VSTs comprising the OOI. Optionally,this feedback may be derived based on indications of how other occupantsfelt about the VST (when it depicted certain content corresponding toOOIs). For example, points of interest may be determined according toaffective crowd data, determine based on how other occupants felt aboutwhat they saw, and optionally find places where the crowd affectiveresponse was positive (e.g., blooming of flowers, etc.). Optionally,some occupants who watched the video related to the unhindered FOV forlong periods (or have a physical window) may be monitored to see when aninteresting view is encountered. Points of interest may involve itemsidentified in the exterior environment using image analysis, includingpresenting the VST when one or more of the following happens: passing afancy vehicle, passing someone the occupant knows, passing a vehicleaccident, passing a police officer, having a sunset view, and passing ananimal the occupant is interested in.

In one embodiment, the system may receive an indication that theoccupant is feeling at least one of nauseous and claustrophobic, andconsequently present the VST in order to help the occupant to confrontthe feeling. For example, when there is a serious spike (e.g., increasein heart-rate and sweating), then the system may present the VSTautomatically. The system may also present a “normal” interior to theoccupant (e.g., natural lighting settings). Optionally, when digitalcontent is consumed (e.g., a game or a movie), the system may checkwhether the affective spike (e.g., increased excitement) is due to thecontent or due to the experience of being in the vehicle with AR or VR.If it is the former case, then the system might not present the VST.

Some aspects of this disclosure involve an autonomous on-road vehiclethat includes a nontransparent Shock-Absorbing Energy DissipationPadding (SAEDP) that is coupled to the compartment of the vehicle and islocated, during normal driving, at eye level in front of an occupant whosits in a front seat of the vehicle. Additionally, the vehicle includesa stiff element that is configured to support the SAEDP and to resistdeformation during collision in order to reduce compartment intrusion.The stiff element is located, during normal driving, at eye levelbetween the SAEDP and the outside environment. Thus, the combination ofthe SAEDP and the stiff element offers the occupant an increased levelof protection, e.g., in the case of a collision, compared to a vehiclein which a traditional window is in place at eye level in the front ofthe vehicle. However, due to it being nontransparent, placing the SAEDPat eye level may obstruct the occupant's view to the outside. Thus, inorder to offer the occupant such a view, in some embodiments, thevehicle also includes a camera configured to take video of the outsideenvironment in front of the occupant, and a computer configured togenerate, based on the video, a representation of the outsideenvironment in front of the occupant at eye level. This representationmay be provided to the occupant using various types of displays.

One non-limiting advantage of the vehicle described above is that itincreases the safety of the occupant in the case of a collision, withoutprohibiting the occupant from obtaining a view of the outsideenvironment.

In one embodiment, an autonomous on-road vehicle includes a compartment,which one or more occupants may occupy while traveling in the vehicle(e.g., by sitting in seats). Coupled to the front of the compartment isa Shock-Absorbing Energy Dissipation Padding (SAEDP) and a stiff elementthat supports the SAEDP. Optionally, the SAEDP is nontransparent. Thestiff element is located, during normal driving, at eye level betweenthe SAEDP and the outside environment. Additionally, the vehicleincludes a camera (e.g., camera 142 or structure 147 that comprisesmultiple cameras), which is configured to take video of the outsideenvironment in front of the occupant, and a computer (e.g., computer143) that is configured to generate, based on the video, arepresentation of the outside environment in front of the occupant ateye level. Optionally, the camera, and/or each of the cameras in thestructure 147, may be based on at least one of the following sensors: aCCD sensor, a CMOS sensor, a near infrared (NIR) sensor, an infraredsensor (IR), and a device based on active illumination such as a LiDAR.Optionally, when the camera comprises multiple cameras, the multiplecameras are directed to multiple directions around the vehicle, and themultiple cameras support generating multiple representations of theoutside environment from different points of view.

It is to be noted that in some embodiments, the SAEDP may be fixed atits location both in normal driving and in times that are not consideredto correspond to normal driving, while in other embodiments, the SAEDPmay change its location during at least some of the times that do notcorrespond to normal driving.

The SAEDP is coupled to the compartment in such a way that it islocated, during normal driving, at eye level in front of an occupant whosits in a front seat of the vehicle. Different types of SAEDPs may beutilized in different embodiments.

In one embodiment, the SAEDP comprises a passive material that is lessstiff than a standard automotive glass window. The passive material isconfigured to protect the occupant's head against hitting the inner sideof the vehicle compartment during a collision. Optionally, the passivematerial has thickness greater than at least one of the followingthicknesses: 1 cm, 2 cm, 5 cm, 10 cm, 15 cm, and 20 cm. Optionally, thethickness of the passive material may refer to the average thickness ofthe SAEDP across the portion of the SAEDP at eye level. Alternatively,the thickness may refer to the maximal thickness at some position of theSAEDP (which is at least one of the values mentioned above).

In another embodiment, the SAEDP comprises a pneumatic pad that isconfigured to inflate in order to protect the occupant's head againsthitting the inner side of the vehicle compartment during collision. Insome examples, the pneumatic pads may be formed from an elastomericmaterial providing chambers containing air or another gas. Optionally,the chambers are retained in compressed deflated condition until beinginflated by the admission of gas pressure controlled by the vehicle'sautonomous-driving control system that is responsible to estimate theprobability and severity of an imminent collision. Additionally oralternatively, the chambers may be provided with restricted passageslimiting the flow out from the chambers to provide shock-absorbingenergy dissipation to reduce the rebound effect. U.S. Pat. No. 5,382,051discloses examples for pneumatic pads that can be used with some of theembodiments.

In yet another embodiment, the SAEDP comprises an automotive airbag,which is configured to protect the occupant's head against hitting theinner side of the vehicle compartment during collision. In one example,during normal driving, the airbag is in a stowed state. The airbag iscoupled to an inflator configured to inflate the airbag with gas to aninflated state, upon receiving an indication indicative of a probabilityof an impact of the vehicle exceeding a threshold. In this example, theairbag is located, when in the stowed state, at eye level in front ofthe occupant.

In some embodiments, the compartment may include a door, and the SAEDPis physically coupled to the door from the inside, such that the SAEDPmoves with the door as the door opens and/or closes.

In some embodiments, the vehicle may include a second SAEDP coupled tothe outer front of the vehicle to minimize damage to a pedestrian duringa pedestrian-vehicle collision.

In one embodiment, the stiff element that supports the SAEDP isnontransparent. In another embodiment, the stiff element may beautomotive laminated glass or automotive tempered glass. Optionally, thestructure of the vehicle comprises a crumple zone located at eye levelbetween the stiff element and the outside environment.

The representation of the outside environment is intended to provide theoccupant with some details describing the outside environment. In someembodiments, the representation of the outside environment is generatedfrom the point of view of the occupant, and it represents how a view ofthe outside environment would look like to the occupant, had there beena transparent window at eye level instead of the SAEDP and/or the stiffelement. Optionally, a display is utilized to present the representationto the occupant.

Various types of displays may be utilized in different embodiments topresent the representation of the outside environment to the occupant.In one embodiment, the display is comprised in an HMD, and the vehiclefurther comprises a communication system configured to transmit therepresentation to the HMD. For example, the HMD may be a virtual realitysystem, an augmented reality system, or a mixed-reality system. In oneembodiment, the display is supported by at least one of the SAEDP andthe stiff element. For example, the display is physically coupled to theSAEDP and/or the stiff element. Optionally, the display is a flexibledisplay. For example, the flexible display may be based on at least oneof the following technologies and their variants: OLED, organic thinfilm transistors (OTFT), electronic paper (e-paper), rollable display,and flexible AMOLED. In one example, the display is flexible enough suchthat it does not degrade the performance of the SAEDP by more than 20%during a collision. In one example, the performance of the SAEDP ismeasured by hitting a crash test dummy head against the SAEDP andmeasuring the head's deceleration using sensors embedded in the head.

FIG. 21a , FIG. 21b , and FIG. 22 illustrate various embodiments of thevehicle described above. Each of the illustrated vehicles comprises across-section view of the vehicles, where each includes a compartment145 for a single occupant (in FIG. 21b ) or more (in FIG. 21a and FIG.22). In the figures, much of the compartment is covered with the SAEDP140, which is nontransparent and comprises a soft passive material(cushiony in its nature). Supporting the SAEDP 140 is a stiff element141, which in the illustrations comprises portions of the exterior(hull) of the vehicle which may optionally be made of one or more of thefollowing materials: fiber-reinforced polymer, carbon fiber reinforcedpolymer, steel, and aluminum. The vehicles also include a camera (suchas camera 142 and/or structure 147 that houses multiple cameras), whichis positioned to capture a front view of the outside environment of thevehicle. Additionally, the vehicles include a computer 143, which may bepositioned in various locations in the vehicle. In some embodiments, thecomputer may be comprised of multiple processors and/or graphicsprocessors that may be located at various locations in the vehicle.

The figures illustrate various types of displays that may be utilized topresent the occupant with the representation of the outside environmentgenerated by the computer 143 based on the video taken by the camera142. In FIG. 21a the representation is presented via an HMD 144, whichmay be, for example, a virtual reality HMD. In FIG. 21b therepresentation is presented via an HMD 146, which may be, for example, amixed-reality headset. And in FIG. 22 the representation may be providedvia one or more of the displays 150, which are coupled to thecompartment. It is to be noted that in the figures described above notall of the described elements appear in each figure.

The figures also illustrate various structural alternatives that may beimplemented in different embodiments described herein. For example, FIG.21a illustrates a vehicle that includes window 148, which may optionallybe an automotive tempered glass window, located in a location in whichthe head of a belted occupant is not expected to hit during collision.FIG. 21b illustrates a vehicle that includes crumple zone 149, which islocated at the front of the vehicle at eye level. The figure alsoillustrates the structure 147 that houses multiple cameras directed tomultiple directions around the vehicle.

In some embodiments, the representation of the outside environment maybe manipulated in order to improve how the outside environment looks tothe occupant. Optionally, this may be done utilizing the computer. Inone example, manipulating the representation includes at least one ofthe following manipulations: converting captured video of an overcastday to video of a sunny day by preserving main items in the capturedvideo (such as vehicles and buildings) and applying effects of a sunnyday, converting unpleasant environment to a nice one, convertingstandard vehicles to futuristic or old fashion vehicles, and adding fansstanding outside and waiving to the occupant.

In one embodiment, the manipulation maintains the main items in theenvironment, such that the occupant would still know from themanipulated representation where he/she is traveling. In anotherembodiment, the manipulated representation maintains the main objects inthe video of the outside environment, such that the main objectspresented in the manipulated video essentially match the main objectsthat would have been seen without the manipulation.

In some embodiments, the vehicle compartment may include an automotivelaminated glass window or automotive tempered glass window located in alocation where the head of a belted occupant is not expected to hit as aresult of collision while traveling in velocity of less than 50 km/h, asillustrated by the dotted rectangle 148 in FIG. 21 a.

In one embodiment, the structure of the vehicle is such that a crumplezone is located at eye level between the stiff element and the outsideenvironment.

Various types of vehicles may benefit from utilization of thenontransparent SAEDP supported by the stiff element and in conjunctionwith the camera and computer, as described above. The following are someexamples of different characterizations of vehicles in differentembodiments. In one embodiment, the vehicle weighs less than 1,500 kgwithout batteries, and it is designed to carry up to five occupants. Inanother embodiment, the vehicle weighs less than 1,000 kg withoutbatteries, and it comprises an engine that is able to sustaincontinuously at most 80 horsepower. In yet another embodiment, thevehicle weighs less than 1,000 kg and it is designed to carry up to twooccupants. In still another embodiment, the vehicle weighs less than 800kg without batteries, and it comprises an engine that is able to sustaincontinuously at most 60 horsepower. In yet another embodiment, thevehicle weighs less than 500 kg without batteries and it comprises anengine that is able to sustain continuously at most 40 horsepower. Andin still another embodiment, the vehicle weighs less than 400 kg withoutbatteries and is designed to carry up to two occupants.

Some aspects of this disclosure involve an autonomous on-road vehiclethat includes a nontransparent Shock-Absorbing Energy DissipationPadding (SAEDP) that is coupled to the compartment and is located,during normal driving, at eye level to the left of the occupant who sitsin a front seat of the vehicle (in the left front seat when the vehiclehas more than one front seat). The SAEDP obstructs at least 30 degreesout of the horizontal unaided field of view (FOV) to the outsideenvironment to the left of the occupant at eye level. Additionally, thevehicle includes a stiff element that is configured to support the SAEDPand to resist deformation during collision in order to reducecompartment intrusion. The stiff element is located, during normaldriving, at eye level between the SAEDP and the outside environment.Thus, the combination of the SAEDP and the stiff element offers theoccupant an increased level of protection, e.g., in the case of acollision to the left side of the vehicle, compared to a vehicle inwhich a traditional window is in place at eye level on the left side ofthe vehicle. However, due to it being nontransparent, placing the SAEDPat eye level may obstruct the occupant's view to the outside. Thus, inorder to offer the occupant such a view, in some embodiments, thevehicle also includes a camera configured to take video of the outsideenvironment to the left of the occupant, and a computer, which isconfigured to generate, based on the video, a representation of theoutside environment to the left of the occupant at eye level. Thisrepresentation may be provided to the occupant using various types ofdisplays.

One non-limiting advantage of the vehicle described above is that itincreases the safety of the occupant in the case of a side collision,without prohibiting the occupant from obtaining a side view of theoutside environment.

In one embodiment, an autonomous on-road vehicle includes a compartment,which one or more occupants may occupy while traveling in the vehicle(e.g., by sitting in seats). Coupled to the compartment is aShock-Absorbing Energy Dissipation Padding (SAEDP) and a stiff elementthat supports the SAEDP. Optionally, the SAEDP is nontransparent. TheSAEDP is located, during normal driving, at eye level to the left of theoccupant who sits in a front seat of the vehicle. The stiff element islocated, during normal driving, at eye level between the SAEDP and theoutside environment. Optionally, the stiff element is nontransparent.Optionally, the stiff element may be automotive laminated glass orautomotive tempered glass.

The vehicle also includes a camera (such as camera 161) that isconfigured to take video of the outside environment to the left of theoccupant, and a computer that is configured to generate, based on thevideo, a representation of the outside environment to the left of theoccupant at eye level. Optionally, the camera comprises multiple camerasdirected to multiple directions around the vehicle, and the multiplecameras support generating multiple representations of the outsideenvironment from different points of view.

FIG. 23 illustrates one embodiment of the autonomous on-road vehicledescribed above, which shows how an SAEDP protects the occupant during acollision. In the figure, SAEDP 160 (which may comprise a passivematerial) is coupled to the stiff element 141. When another vehiclecollides with the side of the vehicle, the occupants head strikes thesoft SAEDP 160, instead of a glass window (which would be positionedthere in many conventional vehicles).

The SAEDP is coupled to the compartment in such a way that it islocated, during normal driving, at eye level to the left of the occupantwho sits in a front seat of the vehicle. Optionally, due to itslocation, the SAEDP obstructs at least 30 degrees out of the horizontalunaided field of view (FOV) to the outside environment to the left ofthe occupant at eye level. Optionally, the SAEDP obstructs at least 45degrees or at least 60 degrees out of the horizontal unaided FOV to theoutside environment to the left of the occupant at eye level. In oneexample of a standard vehicle, such as Toyota Camry model 2015, thefrontal horizontal unaided FOV extends from the left door through thewindshield to the right door.

In some embodiments, the SAEDP is fixed to the left door of the vehicle.In one embodiment, the vehicle has a single seat (occupied by theoccupant). In another embodiment, the vehicle has two or more frontseats and the occupant occupies the leftmost of the two or more frontseats.

Different types of SAEDPs may be utilized in different embodiments. Inone embodiment, the SAEDP comprises a passive material, which is lessstiff than a standard automotive glass window, having a thicknessgreater than at least one of the following thicknesses: 1 cm, 2 cm, 5cm, 10 cm, 15 cm, and 20 cm. In other embodiments, the SAEDP may includean automotive airbag or a pneumatic pad that is configured to inflate inorder to protect the occupant's head against hitting the inner side ofthe vehicle compartment during collision.

In a similar fashion to how the SAEDP and stiff element are utilized tohelp protect the left side of the occupant, the same setup may beapplied to the right side of the vehicle, in order to help protect thatside. Thus, in some embodiments, the vehicle may further include asecond SAEDP located at eye level to the right of the occupant who sitsin the front seat, and a second stiff element located at eye levelbetween the second SAEDP and the outside environment. Optionally, thesecond SAEDP obstructs at least 20 degrees out of the horizontal unaidedFOV to the outside environment to the right of the occupant at eyelevel, and the computer is further configured to generate a secondrepresentation of the outside environment to the right of the occupant.

Some aspects of this disclosure involve an autonomous on-road vehiclethat includes a nontransparent Shock-Absorbing Energy DissipationPadding (SAEDP) that can cover a side window that enables an occupant ofthe vehicle to see the outside environment. The side window is locatedat eye level of an occupant who sits in the vehicle. A motor isconfigured to move the SAEDP over a sliding mechanism between first andsecond states multiple times without having to be repaired. A processoris configured to receive, from an autonomous-driving control system, anindication that a probability of an imminent collision reaches athreshold, and to command the motor to move the SAEDP from the firststate to the second state. In the first state the SAEDP does not blockthe occupant's eye level view to the outside environment, and in thesecond state the SAEDP blocks the occupant's eye level view to theoutside environment in order to protect the occupant's head againsthitting the side window during collision.

One non-limiting advantage of the vehicle described above is that itincreases the safety of the occupant in the case of a side collision.

In one embodiment, an autonomous on-road vehicle includes a side window170, a nontransparent SAEDP (e.g., SAEDP 171), a motor 172, and aprocessor 175.

The processor 175 is configured to receive, from an autonomous-drivingcontrol system (such as autonomous-driving control system 65), anindication indicating that a probability of an imminent collisionreaches a threshold, and to command the motor 172 to move the SAEDP 171from the first state to the second state. In the first state the SAEDP171 does not block the occupant's eye level view to the outsideenvironment, and in the second state, the SAEDP 171 blocks theoccupant's eye level view to the outside environment in order to protectthe occupant's head against hitting the side window during collision.Optionally, the processor is configured to command the motor 172 tostart moving the SAEDP 171 to the second state at least 0.2 second, 0.5second, 1 second, or 2 seconds before the expected time of thecollision.

The motor 172 is configured to move the SAEDP 171 over a slidingmechanism 173 between first and second states multiple times withouthaving to be repaired. For example, during the same voyage, the SAEDP171 may go up and down multiple times without a need for the occupant oranyone else to repair the SAEDP 171 and/or other components (such asmotor 172 or the window 170) in order to the SAEDP 171 to be able tocontinue its operation correctly (i.e., continue moving up and down whenneeded). In some examples, the motor 172 is a motor designed to move theSAEDP 171 more than twice, more than 100 times, and/or more than 10,000times without being replaced.

The side window 170 is located at eye level of an occupant who sits inthe vehicle, which enables the occupant to see the outside environment.In one embodiment, the side window 170 is a power window. In thisembodiment, the power window comprises a window regulator that transferspower from a window motor 177 to the side window glass in order to moveit up or down. The motor 172 is coupled to an SAEDP regulator thattransfers power from the motor 172 to the SAEDP 171 in order to move itup or down. In this embodiment, the SAEDP regulator is located closer tothe inner side of the compartment then the window regulator. Optionally,the motor 172 and the window motor 177 may be of the same type or ofdifferent types.

In one embodiment, the SAEDP 171 comprises a passive material, which isless stiff than a standard automotive glass window, having thicknessgreater than at least one of the following thicknesses: 1 cm, 2 cm, 5cm, 10 cm, 15 cm, and 20 cm. Optionally, the vehicle may include astorage space in a door of the vehicle, which is configured to store theSAEDP 171 in the first state. Additionally or alternatively, the vehiclemay include a storage space in the roof of the vehicle, which isconfigured to store the SAEDP 171 in the first state.

Optionally, the SAEDP 171 may move upwards when switching between thefirst and second states, and the top of the SAEDP has a profile (such asa triangle or a quarter sphere) which reduces the risk of catching the apart of the occupant (e.g., a finger or limb) or the occupant'sclothing, between the top of the SAEDP 171 and an upper frame whenmoving the SAEDP 171 to the second state.

In one embodiment, when switching the SAEDP 171 quickly between thefirst and second states, the SAEDP 171 is configured not to cover arange of 1 to 5 cm of the top height of the window. Optionally, keepingsaid range unoccupied reduces the risk of catching the occupant'sfingers or limb by the edge of the SAEDP 171 when moving the SAEDP 171to the second state.

In some embodiments, the vehicle may include additional SAEDPs thatcover additional regions of the vehicle's compartment (besides the sidewindow 170). In one example, the vehicle includes an SAEDP 176 thatcovers at least a portion of the roof of the vehicle.

In some embodiments, the vehicle includes a camera (e.g., camera 178 a),which is configured to take video of the outside environment while theSAEDP 171 is in the second state. Additionally, in these embodiments,the vehicle may include a computer (such as computer 13), which isconfigured to generate a representation of the outside environment basedon the video, and a display configured to present the representation ofthe outside environment to the occupant. The display may be physicallycoupled to the compartment and/or belong to an HMD. Optionally, thecamera is fixed to the SAEDP 171, and thus moves along with the SAEDP171 when it is moved between the first and second states. Optionally,the display is fixed to the SAEDP 171, and thus moves along with theSAEDP 171 when it is moved between the first and second states.Optionally, the display is configured to show, at eye level, arepresentation of the outside environment when the SAEDP is in thesecond state. In one example, the display is a flexible display. Inanother example, the camera comprises multiple cameras directed tomultiple directions around the vehicle, and the computer is configuredto generate at least two different representations of the outsideenvironment, from at least two different points of view, for twooccupants who sit in the vehicle.

In some embodiments, in addition to raising the SAEDP 171, one or moreof the displays mentioned above is utilized to present the occupant avideo of the threat and the predicted trajectory that could result inthe collision, in order to explain why the SAEDP 171 is being moved tothe second state.

FIG. 24a and FIG. 24b illustrate an example of a vehicle in which theside window may be covered by an SAEDP that can move up and down. Thefigures illustrate cross-sections of the vehicle, which show how theSAEDP 171 may move from the first state (in FIG. 24a ) to the secondstate (in FIG. 24b ). The dotted line 179 indicates that the SADEP 171does not close the entire gap over the window (e.g., in order to avoidcatching the occupant's hair). The figures also illustrate slidingmechanism 173, which may be utilized to guide and assist in the movementof SAEDP 171. FIG. 24b also illustrates camera 178 a and display 178 b,which are connected to a processor that generates, based on the videoreceived from the camera, a view of the outside environment when anSAEDP (on the right side of the vehicle) is in the second state. Theview of the outside environment is presented to the occupant on thedisplay 178 b. Camera and display on the left SAEDP 171, whichcorrespond to camera 178 a and display 178 b, are not shown in thefigure in order to make it clearer; however it is to be understood thatsuch camera and display may be fitted to any relevant moving SAEDP.

In one embodiment, the mechanism that moves the SAEDP 171 between thefirst and second states (referred to as the “SAEDP mechanism”) issimilar to a power window regulator that moves an automobile window upand down. As with automobile power windows, the SAEDP regulator may bepowered by an electric motor, which may come with the SAEDP regulator asone unit, or as a system that enables the motor or regulator to bereplaced separately. The SAEDP mechanism includes a control system, amotor, a gear reduction, a sliding mechanism and the SAEDP, which areusually fixed on the door, but may alternatively be fixed on the roof asdisclosed below. The sliding mechanism may have different architectures,such as Bowden type, double Bowden type, cable spiral, or crossedlevers.

In a first example, the SAEDP mechanism is similar to a double Bowdenpower window mechanism, in which the SAEDP 171 is fixed on two supportsrespectively constrained along two rails. The control system drives themotor that wraps two Bowden cables, which move two supports and,consequently, the SAEDP 171. A Bowden cable transmits mechanical forcethrough the movement of an inner cable relative to an outer housing, andin the case of a DC motor, the basic operations of the motor areaccomplished by reversing the polarity of its power and ground input.

In a second example, the SAEDP mechanism is similar to a gear-drive typepower window regulator; in this case, the SAEDP mechanism includes anSAEDP motor to power the mechanism, gear drive and geared arm to movethe SAEDP 171 between the first and second states, and an SAEDP holdingbracket to hold the SAEDP 171.

In a third example, the SAEDP mechanism is similar to a cable type powerwindow regulator; in this case, the SAEDP regulator includes an SAEDPmotor that drives a wire cable though a mechanism, a series of pulleysthat guide the cable, and a regulator carriage attached to the cable andto the SAEDP 171 and slides on the regulator track. One or more tracksmay be mounted vertically inside the door panel that serves as a guidepiece when the SAEDP 171 slides up and down. Depending on the design,the setup may have one main regulator track in the center of the door,or have a track on each side of the SAEDP.

In a fourth example, the SAEDP regulator is similar to a scissor powerwindow regulator; in this case, a motor operates a gear wheel thatraises and lowers the SAEDP 171 by the use of a scissor action of rigidbars.

The motor that moves the SAEDP 171 over the sliding mechanism may be anysuitable motor, such as a DC electric motor, an AC electric motor, or apneumatic motor.

In one embodiment, the indication that the probability of an imminentcollision reaches a threshold is received from the autonomous-drivingcontrol system 65 that calculates the probability based on thetrajectory of the vehicle and information about the road. Optionally,the information about the road may be received from one of more of thefollowing sources: a sensor mounted to the vehicle, a sensor mounted ona nearby vehicle, a road map, a stationary traffic controller nearby thevehicle, and a central traffic controller that communicates with thevehicle via wireless channel.

In one embodiment, the processor 175 is further configured to receive anupdated indication that the probability of the imminent collision doesnot reach a second threshold, and to command the motor to move the SAEDPto the first state. In this embodiment, the second threshold denotes aprobability for a collision that is equal or lower than the threshold.

FIG. 25a and FIG. 25b illustrate a motor configured to move a frontnontransparent Shock-Absorbing Energy Dissipation Padding (SAEDP)between first and second states.

Some aspects of this disclosure involve an autonomous on-road vehiclethat includes an outer nontransparent Shock-Absorbing Energy DissipationPadding (SAEDP) mounted to the front side of the vehicle during normaldriving, such that the SAEDP is in front of and at eye level of anoccupant who sits in a front seat of the vehicle. Additionally, thevehicle includes a camera that is mounted to the vehicle and isconfigured to take video of the outside environment in front of theoccupant, and a computer configured to generate, based on the video, arepresentation of the outside environment at eye level for the occupant.

One non-limiting advantage of the vehicle described above is that itincreases the safety of a pedestrian in case of a vehicle-pedestriancollision, without prohibiting the occupant of the vehicle fromreceiving a frontal view of the outside environment.

FIG. 26a illustrates one embodiment of an autonomous on-road vehiclethat includes outer nontransparent SAEDP 190, which is mounted to thefront side of the vehicle during normal driving, such that the SAEDP 190is in front of and at eye level of an occupant who sits in a front seatof the vehicle. The SAEDP 190 is less stiff than a standard automotiveglass window and is designed to absorb some of the crashing energytransmitted to a pedestrian during a pedestrian-vehicle collision.Additionally, the vehicle includes a camera (such as camera 142), whichis mounted to the vehicle and is configured to take video of the outsideenvironment in front of the occupant, and a computer (such as computer143), which is configured to generate, based on the video, arepresentation of the outside environment at eye level for the occupant.Optionally, the representation is generated from the point of view ofthe occupant. Optionally, the vehicle includes a display configured topresent the representation to the occupant. For example, the display maybelong to an HMD worn by the occupant. In another example, the displaymay be coupled to the compartment of the vehicle, and may be a flexibledisplay.

The SAEDP 190 may be implemented utilizing various approaches indifferent embodiments described herein. In one embodiment, the SAEDP 190comprises a passive material. Optionally, the SAEDP 190 has thicknessgreater than at least one of the following thicknesses: 1 cm, 2 cm, 5cm, 10 cm, 15 cm, and 20 cm.

In another embodiment, the SAEDP 190 comprises an automotive airbagconfigured to inflate in order to protect the pedestrian. FIG. 26billustrates an outer SAEDP 190 that includes two air bags 192 configuredto absorb some of the crashing energy transmitted to a pedestrian duringa pedestrian-vehicle collision. Optionally, the airbag has a stowedcondition and an inflated condition. The airbag is coupled to aninflator configured to inflate the airbag with gas, and the airbag islocated, in the stowed condition, at eye level in front of the occupant.In this embodiment, the vehicle further includes an autonomous-drivingcontrol system, such as autonomous-driving control system 65, which isconfigured to calculate a probability of pedestrian-vehicle collision,based on measurements of sensors mounted to the vehicle, and to commandthe airbag to inflate before the pedestrian head hits the vehicle.

In yet another embodiment, the SAEDP 190 comprises a pneumatic padconfigured to inflate in order to protect the pedestrian. In thisembodiment, the vehicle further includes an autonomous-driving controlsystem, such as autonomous-driving control system 65, which isconfigured to calculate a probability of pedestrian-vehicle collision,based on measurements of sensors mounted to the vehicle, and to commandthe pneumatic pad to start inflate at least 0.5 second before theexpected time of the collision in order to protect the pedestrian.Optionally, the pneumatic pad is reusable, and can be used multipletimes without the need to be repaired. For example, the vehiclecomprises a mechanism to deflate and/or stow the pneumatic pad, withoutrequiring its repair and/or replacement.

Some aspects of this disclosure involve an autonomous on-road vehiclethat includes a window located at eye level of an occupant who sits in afront seat of the vehicle (e.g., a windshield), a reusablenontransparent Shock-Absorbing Energy Dissipation Padding (SAEDP), amotor, and a processor. The window enables the occupant to see theoutside environment. The motor is configured to move the SAEDP over asliding mechanism between first and second states multiple times withouthaving to be repaired. In the first state the SAEDP does not block theoccupant's eye level frontal view to the outside environment, and in thesecond state the SAEDP blocks the occupant's eye level frontal view tothe outside environment. Additionally, in the second state the SAEDP canabsorb some of the crashing energy transmitted to a pedestrian during apedestrian-vehicle collision. The processor is configured to receive,from an autonomous-driving control system, an indication indicative ofwhether a probability of an imminent pedestrian-vehicle collisionreaches a threshold. Responsive to receiving an indication of animminent collision (e.g., within less than 2 seconds), the processor isconfigured to command the motor to move the SAEDP from the first stateto the second state.

One non-limiting advantage of the vehicle described above is that itincreases the safety of a pedestrian in case of a vehicle-pedestriancollision, without prohibiting the occupant of the vehicle fromreceiving a frontal view of the outside environment during normaldriving.

FIG. 27a and FIG. 27b illustrate a motorized external SAEDP 121 that canmove between first and second states multiple times. The figuresillustrate how the SAEDP 121 can move from the first state (in FIG. 27a) to the second state (FIG. 27b ) by having the motor 122 move the SAEDP121 over sliding mechanism 123. Additionally, the figures illustrateoptional camera 126 that is embedded in the SAEDP 121, and which mayprovide video to a processor configured to generate a representation ofthe outside environment when the SAEDP 121 is in the second state.

In one embodiment, an autonomous on-road vehicle includes window 120,reusable SAEDP 121, motor 122, and processor 124. The window 120, whichis located at eye level of an occupant who sits in a front seat of thevehicle, and which may be a windshield, enables the occupant to see theoutside environment. The SAEDP 121 is reusable, i.e., it may be movedmultiple times without the need to replace it or repair it after eachuse. The SAEDP 121 may be implemented utilizing various approaches indifferent embodiments described herein. In one embodiment, the SAEDP 121comprises a passive material. Optionally, the SAEDP 121 has thicknessgreater than at least one of the following thicknesses: 1 cm, 2 cm, 5cm, 10 cm, 15 cm, and 20 cm. In another embodiment, the SAEDP 121comprises a pneumatic pad configured to inflate in order to protect thepedestrian. Optionally, the pneumatic pad is reusable, and the processor124 is configured to command the pneumatic pad to start inflate at least0.5 second before the expected time of the pedestrian-vehicle collision.

The motor 122 is configured to move the SAEDP 121 over a slidingmechanism 123 between first and second states multiple times withouthaving to be repaired. In the first state the SAEDP 121 does not blockthe occupant's eye level frontal view to the outside environment, and inthe second state the SAEDP 121 blocks the occupant's eye level frontalview to the outside environment. When in the second state, the SAEDP 121is configured to absorb some of the crashing energy transmitted to apedestrian during a pedestrian-vehicle collision.

The processor 124 is configured to receive, from an autonomous-drivingcontrol system (such as autonomous-driving control system 65), anindication indicative of whether a probability of an imminentpedestrian-vehicle collision reaches a threshold. Optionally, most ofthe time the vehicle travels, the processor 124 does not provide anindication that the probability reaches the threshold. Responsive toreceiving an indication of an imminent collision (e.g., within less than2 seconds), the processor 124 is configured to command the motor 122 tomove the SAEDP 121 from the first state to the second state. Optionally,the processor 124 is configured to command the motor to start moving theSAEDP 121 to the second state at least 0.2 second, 0.5 second, 1 second,or 2 seconds before the pedestrian-vehicle collision in order to protectthe pedestrian.

In one example, the vehicle includes a sensor configured to detect thedistance and angle between the vehicle and a pedestrian, and theautonomous-driving control system calculates the probability of theimminent pedestrian-vehicle collision based on the data obtained fromthe sensor, the velocity of vehicle and the possible maneuver.

In one embodiment, the processor 124 is further configured to receive anupdated indication that indicates the probability of the imminentpedestrian-vehicle collision does not reach a second threshold, and tocommand the motor 122 to move the SAEDP to the first state. Optionally,the second threshold denotes a probability for a pedestrian-vehiclecollision that is equal or lower than the threshold.

In one embodiment, the vehicle includes a camera (such as camera 126),which is configured to take video of the outside environment while theSAEDP 121 is in the second state. Additionally, in this embodiment, thevehicle may further include a computer configured to generate, based onthe video, a representation of the outside environment, and a displayconfigured to present the representation of the outside environment tothe occupant while the SAEDP 121 is in the second state. Optionally, thecamera 126 is fixed to the SAEDP 121 from the outer side, and thus moveswith the SAEDP 121 when it moves between the first and second states.Optionally, the display is fixed to the SAEDP 121 from the inner side,and thus also moves with the SAEDP 121 when it moves between the firstand second states; the occupant can see the display via the window 120when the SAEDP is in the second state. In an alternative embodiment, thedisplay is physically coupled to the compartment (such as a windshieldthat also functions as a display) and/or comprised in an HMD worn by theoccupant.

In addition to, or instead of, moving an SAEDP when a collision isimminent, in some embodiments, an imminent collision may prompt a raiseof one or more power windows in the vehicle. This approach isillustrated in the following embodiment.

In one embodiment, a safety system for an occupant of an autonomouson-road vehicle includes an automobile power window and anautonomous-driving control system (such as autonomous-driving controlsystem 65). The autonomous-driving control system is configured tocalculate probability of an imminent collision based on data receivedfrom sensors coupled to the vehicle. Responsive to detecting that theprobability reaches a threshold, and at least one second before theexpected collision, the autonomous-driving control system commands thepower window to rise. In this embodiment, a raised power window providesan improved safety for the occupant of the vehicle during collisioncompared to a lowered power window. Optionally, responsive to detectingthat the probability reaches the threshold, and at least two secondsbefore the expected collision, the autonomous-driving control system isconfigured to command the power window to rise.

In one embodiment, the vehicle further includes an SAEDP coupled to thepower window from the inside (e.g., as illustrated in FIG. 24a and FIG.24b ).

In one embodiment, the power window is nontransparent, and the safetysystem for the occupant includes a display coupled to the power windowfrom the inside. The display is configured to present to the occupant avideo see-trough (VST) of the outside environment that is generatedbased on a video camera pointed at the outside environment.

In another embodiment, the power window is made of a nontransparentmaterial, which is stronger than a standard automotive tempered glasshaving the same dimensions. In this embodiment, the vehicle furthercomprises a camera configured to take video of the outside environment(such as camera 178 a), and a computer configured to generate a VST tothe outside environment based on the video. Optionally, the camera isphysically coupled to the power window, such that the camera is raisedand lowered with the power window. Additionally or alternatively, thecamera may be coupled to an element of the vehicle that is not the powerwindow, such that the camera does not move up and down when the powerwindow is raised and lowered. Optionally, the vehicle further includes adisplay coupled to the inner side of the power window, such that thedisplay is at eye level when the power window is raised, and below eyelevel when the power window is lowered; the display is configured topresent the VST to the occupant. Optionally, the occupant wears an HMD,and the computer is configured to present the VST to the occupant on theHMD when the power window is raised, and not to present the VST to theoccupant on the HMD when the power window is lowered.

With autonomous vehicles, it is not necessary for occupants to sit. Someaspects of this disclosure involve autonomous vehicles in which anoccupant of the vehicle may lay down. Such a design for an autonomousvehicle has the advantage of enabling a more comfortable position forcertain activities (e.g., relaxing or sleeping). Additionally, such avehicle may be built to be lower than traditional vehicles, which canoffer advantages in terms of increased safety (e.g., a lower center ofgravity offers better stability) and better vehicle aerodynamics.

In order to increase the safety of the occupant, various forms ofpadding may be used in the compartment of the vehicle. In someembodiments, an autonomous on-road vehicle designed for lying downincludes a closed compartment and a mattress, having an averagethickness of at least 3 cm, which covers at least 50% of the compartmentfloor. In the compartment, there is a nontransparent Shock-AbsorbingEnergy Dissipation Padding (SAEDP), having an average thickness of atleast 1 cm, which covers at least 50% of the compartment side walls andat least 60% of the compartment front wall during normal driving.Optionally, the SAEDP is supported by a stiff element that resistsdeformation during a collision in order to reduce compartment intrusion.Additionally, the vehicle includes a camera configured to take video ofthe outside environment, a computer configured to generate arepresentation of the outside environment based on the video, and adisplay configured to present the representation to the occupant.

In one embodiment, an autonomous on-road vehicle designed for lying downincludes a closed compartment 210, a mattress 211, an SAEDP 212 coveringportions of the compartment 210, a camera (e.g., the structure 147 thathouses multiple cameras), a computer (e.g., the computer 143), and adisplay 215. FIG. 28 illustrates one embodiment of a vehicle compartment210 in which an occupant may lay down. In the figure, the occupant islying down on mattress 211, which covers the floor of the compartment210, and is watching a movie on the display 215. The SAEDP 212 coversthe front, roof, and back of the compartment 210. It is to be noted thatthe SAEDP 212 also covers portions of the side walls of the compartment210, however, this is not illustrated to enable a clearer image of theembodiment. The figure also includes an airbag 216, which may beinflated below the SAEDP 212 in order to protect the occupant andrestrain his/her movement in the case of a collision.

The mattress 211 covers at least 50% of the compartment floor.Optionally, the mattress 211 covers at least 80% of the compartmentfloor. In one embodiment, the mattress 211 has an average thickness ofat least 3 cm. In other embodiments, the average thickness of themattress 211 is greater than at least one of the following thicknesses:5 cm, 7 cm, 10 cm, 20 cm, and 30 cm.

The SAEDP 212 is a nontransparent SAEDP, having an average thickness ofat least 1 cm. Optionally, the SAEDP 212 covers at least 50% of thecompartment side walls and at least 60% of the compartment front wallduring normal driving. In one embodiment, the average thickness of theSAEDP 212 is greater than at least one of the following thicknesses: 2cm, 3 cm, 5 cm, 10 cm, 15 cm, and 20 cm. In another embodiment, theSAEDP 212 covers at least 80% of the compartment side walls and at least80% of the compartment front wall. In yet another embodiment, the SAEDPcovers at least 50% of the compartment roof. In still anotherembodiment, the mattress and the SAEDP cover essentially the entirecompartment interior.

In addition to the SAEDP 212, in some embodiments additional measuresmay be employed in order to improve the safety of the occupant. In oneembodiment, the vehicle includes an automotive airbag configured todeploy in front of the SAEDP 212 in order to protect the occupant, inaddition to the SAEDP 212, against hitting the inner side of the vehiclecompartment during a collision. It is noted that the meaning that theairbag deploys in front of the SAEDP is that the airbag deploys towardsthe inner side of the compartment. Optionally, the airbag has a stowedcondition and an inflated condition, and the airbag is coupled to aninflator configured to inflate the airbag with gas upon computing apredetermined impact severity. The stowed airbags may be stored invarious positions, such as stored essentially in the middle of the frontwall, stored essentially in the middle of the rear wall, stored in theside walls (possibly two or more horizontally spaced airbags), andstored in the roof (possibly one or more airbags towards the front ofthe compartment and one or more airbags towards the rear of thecompartment).

In some embodiments, various additional safety measures may be utilizedto improve the safety of the occupant while traveling, such as asleeping net and/or a safety belt, as described for example in U.S. Pat.Nos. 5,536,042 and 5,375,879.

Stiff element 213 is configured to support the SAEDP 212 and to resistdeformation during a collision in order to reduce compartment intrusion.Part of the stiff element 213 is located, during normal driving, at eyelevel between the SAEDP 212 and the outside environment. Optionally, thestiff element covers, from the outside, more than 80% of the SAEDP onthe compartment side walls. Optionally, the vehicle also includes acrumple zone located at eye level between the stiff element 213 and theoutside environment.

In another embodiment, the vehicle includes a pneumatic pad configuredto inflate in order to protect the occupant, in addition to the SAEDP212, against hitting the inner side of the vehicle compartment during acollision. Optionally, the pneumatic pad is configured to deploy infront of the SAEDP 212 towards the inner side of the compartment.Alternatively, the pneumatic pad is located between the SAEDP 212 andthe stiff element 213, and is configured to deploy behind the SAEDP 212.The pneumatic pad may be mounted to various locations, such as mountedto the front wall, mounted to the rear wall, mounted to the side walls,and/or mounted to the roof.

The camera is configured to take video of the outside environment. Thecomputer is configured to generate a representation of the outsideenvironment based on the video. Optionally, the representation isgenerated from the point of view of the occupant. The display 215 isconfigured to present the representation to the occupant. In oneembodiment, the display 215 is comprised in an HMD, and the vehiclefurther comprises a communication system configured to transmit therepresentation to the HMD. In another embodiment, the display 215 isphysically coupled to at least one of the SAEDP 212 and the stiffelement 213 at eye level of the occupant. Optionally, the display 215 isa flexible display. For example, the display 215 may be a flexibledisplay that is based on at least one of the following technologies andtheir variants: OLED, organic thin film transistors (OTFT), electronicpaper (e-paper), rollable display, and flexible AMOLED. Optionally, thedisplay 215 is flexible enough such that it does not degrade theperformance of the SAEDP by more than 20% during a collision.

Having a vehicle compartment that is designed to allow an occupant tolay down comfortably can be done using various compartment designs,which may be different from the designs used in standard vehicles, inwhich occupants primarily sit up. In one example, the vehicle does nothave an automotive seat with a backrest and safety belt, which enablesthe occupant to sit straight in the front two thirds of the compartment.In another example, the vehicle is designed for a single occupant, andthe average distance between the mattress and the compartment roof isbelow 80 cm. In still another example, the vehicle is designed for asingle occupant, and the average distance between the mattress and thecompartment roof is below 70 cm. In still another example, the vehicleis designed for a single occupant, and the average distance of thecompartment roof from the road is less than 1 meter. And in yet anotherexample the vehicle is designed for a single occupant, and the averagedistance of the compartment roof from the road is less than 80 cm.

It is to be noted that the use of the terms “floor”, “roof”, “sidewalls”, and “front wall” with respect to the compartment are to beviewed in their common meaning when one considers the compartment to bea mostly convex hull in 3D, such as having a shape that resembles acuboid. Thus, for example, an occupant whose face faces forward, willsee the front wall ahead, the floor when looking below, the roof whenlooking above, and a side wall when looking to one of the sides (left orright). In embodiments that do not resemble cuboids, alternativedefinitions for these terms may be used based on the relative region (in3D space) that each of the portions of the compartment occupy. Forexample, the floor of the compartment may be considered to be anyportion of the compartment which is below at least 80% of the volume ofthe compartment. Similarly, the roof may be any portion of thecompartment that is above at least 80% of the volume of the compartment.The front wall may be any portion of the compartment that is ahead of atleast 80% of the volume of the compartment, etc. Note that using thisalternative definitions, some portions of the compartment may becharacterized as belonging to two different regions (e.g., the frontwall and the roof).

Some aspects of this disclosure involve an autonomous on-road vehiclethat includes a compartment having a large mirroring element located infront of an occupant who sits in a front seat of the compartment.Optionally, the mirroring element has height and width exceeding 25×25cm, such that it covers a square that is at least those dimensions.Optionally, the mirroring element captures a region corresponding to atleast 10×10 degrees, including a region spanning from the horizon to 10°below the horizon, of the occupant's forward field of view during normaldriving. Additionally, the mirroring element provides an effect ofreflecting more than 25% of the light arriving from the occupant'sdirection. In some embodiments, the vehicle has an advantage that it canincrease the perceived compartment volume, which can make travelling inthe vehicle more pleasurable for some people.

FIG. 29 illustrates one embodiment of a vehicle having a front mirroringelement. The figure illustrates how the occupant 222 can see herreflection 223 in the mirroring element 220.

In one embodiment, the mirroring element 220 is located in front of anoccupant who sits in a front seat of the vehicle. The mirroring element220 provides an effect of reflecting more than 25% of the light arrivingfrom the occupant's direction. Optionally, the mirroring element 220provides an effect of reflecting more than at least one of 50%, 80%, and90% of the light arriving from the occupant's direction. Optionally, themirroring element 220 increases the volume of the vehicle compartment asperceived by the occupant due to the reflection effect.

In one embodiment, the mirroring element 220 has height and widthexceeding 25×25 cm, such that it covers a square that is at least thosedimensions. Optionally, the mirroring element 220 captures a regioncorresponding to at least 10×10 degrees, including a region spanningfrom the horizon to 10° below the horizon, of the occupant's forwardfield of view during normal driving. Optionally, the height and width ofthe mirroring element 220 is above 50×40 cm, and the mirroring elementcaptures above 30×30 degrees of the occupant's field of view, includinga region spanning from the horizon to 30° below the horizon. Optionally,the mirroring element 220 covers more than 25% of the area where aconventional windshield of a normal non-autonomous on-road vehicle inyear 2016 is expected to be located.

The mirroring element 220 may be implemented in different ways, indifferent embodiments. In one embodiment, the mirroring element 220comprises an optical mirror that is essentially flat and perpendicularto the ground. In this embodiment, most of the effect of reflecting isgenerated by the optical mirror. In one example, an optical mirror thatis essentially flat and perpendicular to the ground refers to an opticalmirror having a radius of curvature greater than a meter and a deviationbelow ±30° from the perpendicular to the ground.

In another embodiment, the mirroring element 220 comprises a Fresneltype optical reflector comprising many reflecting prisms. In thisembodiment, the Fresnel type optical reflector is not flat, but thereflecting prisms are arranged in angles that reflect the light from theoccupant such as to imitate a flat mirror.

In one embodiment, the mirroring element 220 comprises an electronicdisplay that operates based on a camera 221 configured to take video ofthe occupant 222, and a computer 224 configured to generate a digitalrepresentation of the occupant 223 based on the video of the occupant.In this case, most of the effect of reflecting is generated by lightemitted by the electronic display (which is the output of the video ofthe occupant as generated by the computer 224).

Due to the close proximity between the at least one camera 221 and theoccupant, it may be necessary to stitch the reflecting effect of a largearea from multiple cameras that capture the occupant from differentangles. In one example, the camera 221 comprises first and secondcameras, located to the right and left of the occupant, respectively. Inanother example, the camera 221 comprises first and second cameras,located above the level of the occupant's nose and below the level ofthe occupant's collarbone, respectively, and less than 80 cm away fromthe occupant.

In still another embodiment, the effect of reflecting is achieved by adigital mirror that comprises multiple cameras embedded within thedigital mirror and configured to take images of the occupant, andmultiple light emitting pixels configured to emit light rays thatgenerate the effect of reflecting.

In one embodiment, the mirroring element 220 comprises an electronicdisplay, and the vehicle further includes a camera configured to takevideo of the outside environment in front of the occupant, and acomputer configured to generate, based on the video, a representation ofthe outside environment in front of the occupant at eye level, fordisplaying on the mirroring element 220. In addition to the effect ofreflecting, in some embodiments the mirroring element 220 may also beconfigured to operate in a second state in which it provides an effectof reflecting less than 20% of the light arriving from the occupant'sdirection, and to present to the occupant the representation of theoutside environment. Here, presenting the representation may imitate atransparent windshield to the outside environment.

The vehicle may also include, in some embodiments, a display configuredto present to the occupant, at eye level in front of the occupant, therepresentation of the outside environment instead of, or in addition to,the effect of reflecting. Optionally, the vehicle may include a userinterface configured to enable the occupant to switch between seeing theeffect of reflecting and seeing the representation of the outsideenvironment. In one example, the display is comprised in a head-mounteddisplay (HMD). In another example, the display is mechanically coupledto the compartment, and/or the mirroring element comprises the display.

As described above, the effect of reflecting light that comes from thedirection of the occupant may be obtained, in some embodiments,utilizing an HMD. The following is a more detailed example of such anembodiment.

In one embodiment, an autonomous on-road vehicle includes a compartmentcomprising a front seat, a camera, a computer, a communication module,and an HMD. The camera is physically coupled to the vehicle, configuredto take images of an occupant who sits in the front seat. The computeris configured to generate, based on the images, a video that shows amirror effect. Herein, the “mirror effect” involves presenting theoccupant with an image that is similar to an image that the occupantwould see had there been an actual mirror in front of the occupant.Optionally, the system enables the occupant to have control on thesynthetic mirror effect, such as control on the distance between theoccupant and her image, and/or control on the width of the image. Thecommunication module is configured to transmit the video to ahead-mounted display (HMD) worn by the occupant, and the HMD isconfigured to present the video to the occupant, such that the occupantsees a representation of his or her reflection when looking forward ateye level. Optionally, the HMD is selected from the group comprising: avirtual reality headset, an augmented reality headset, and mixed realityheadset. Optionally, the video presented to the occupant captures atleast 30×30 degrees of the occupant's field of view, spanning at leastfrom the horizon to 30° below the horizon.

In one embodiment, the vehicle described above further includes a secondcamera configured to take a second set of images of the outsideenvironment in front of the occupant. In this embodiment, the computeris further configured to generate a representation of the outsideenvironment based on the second set of images, and the HMD is configuredto present the representation of the outside environment instead of thevideo that shows the mirror effect or in addition to the video thatshows the mirror effect.

In one embodiment, the vehicle described above may include multiplecameras that capture images that are utilized by the computer togenerate the video that shows the mirror effect. In one example, thevehicle comprises at least first and second cameras, which are locatedto the right and left of the occupant, respectively. In another example,the vehicle comprises at least first and second cameras, located abovethe level of the occupant's nose and below the level of the occupant'scollarbone, respectively, and which are less than 80 cm away from theoccupant.

Some aspects of this disclosure involve utilizing one or morenontransparent side beams in order to help protect the occupant in acase of a collision. The one or more side beams are stiffer thanautomotive laminated glass. In one example, a side beam comprises a hightensile steel pipe and pressed material. In another example, a side beamcomprises an aluminum extruded shape.

In one embodiment, an autonomous on-road vehicle that weighs less than1500 kg without batteries includes a nontransparent side beam, which isstiffer than automotive laminated glass, and is located during normaldriving to the left and at eye level of an occupant who sits in a frontseat. The vehicle also includes a camera configured to take video of theoutside environment to the left of the occupant, and a computerconfigured to generate for the occupant, based on the video, arepresentation of the outside environment to the left of the occupant ateye level. Optionally, the representation is presented to the occupantusing a display. In one example, the display is comprised in ahead-mounted display (HMD), and the vehicle further comprises acommunication system configured to transmit the representation to theHMD. In another example, the display is coupled to the inner side of thecompartment.

The following is a description of an embodiment of a vehicle in whichone or more side beams may be utilized to help protect the left side ofa vehicle (and an occupant therein). In one embodiment, an autonomouson-road vehicle includes a nontransparent side beam, a camera (e.g., thecamera 161), and a computer (e.g., the computer 143). Optionally, thevehicle weighs less than 1500 kg without batteries.

The nontransparent side beam, which is stiffer than automotive laminatedglass, is located, during normal driving, at eye level to the left of anoccupant who sits in a front seat of the vehicle. Optionally, the sidebeam is at least 60 cm long and at least 1 cm wide (herein “cm” refersto centimeters). In one example, the side beam comprises a high tensilesteel pipe and pressed material. In another example, the side beamcomprises an aluminum extruded shape.

The camera configured to take video of the outside environment to theleft of the occupant. The computer is configured to generate for theoccupant, based on the video, a representation of the outsideenvironment to the left of the occupant at eye level. Optionally, therepresentation is generated from the point of view of the occupant, andthe vehicle further includes a display configured to present therepresentation to the occupant. In one example, the display is comprisedin a head-mounted display (HMD), and the vehicle further comprises acommunication system configured to transmit the representation to theHMD. In another example, the display is coupled to the inner side of thecompartment.

In some embodiments, a plurality of beams, which may be similar to theside beam described above, may be located in the region of the left sideof the vehicle. It is to be noted that the plurality of beams may notnecessarily all have the same dimensions or be made of the same exactmaterials. Optionally, the plurality of beams may be connected invarious ways in order to form a structure that can better resistcompartment deforming in the case of a collision. For example, in oneembodiment, the vehicle includes an additional side beam that is notparallel to the side beam and crosses it to form an “X” shapedstructure. Herein, an “X” shaped structure refers to any structure oftwo nonparallel beams that cross, and the two do not necessarily form asymmetric “X” (e.g., a symmetric “X” may be obtained when the two beamsare of equal length and cross at their centers). This structure isillustrated in FIG. 30a where beams 231 and 232 cross to form an “X”shape structure. It is to be noted that the beam 231 is at eye level(i.e., at least a portion of the beam 231 is at the same height as atypical occupant's eyes). FIG. 30a illustrates one example of aconfiguration of a plurality of beams located in the left side of thevehicle.

Different designs of vehicles may benefit from utilizing one or moresides beam as described above. In one embodiment, the side beam isembedded in a movable structure that changes its location with respectto the vehicle compartment. For example, the side beam may be part of aleft door of the vehicle. In another embodiment, the side beam isembedded in the vehicle compartment itself and does not change itslocation with respect to the rest of the compartment. For example, theside beam may be placed in a compartment wall to the left of theoccupant.

In order to better protect the occupant from injury due to hitting thehead against the side of the vehicle, in some embodiments, the vehiclemay comprise a nontransparent Shock-Absorbing Energy Dissipation Padding(SAEDP) located at eye level between the side beam and the occupant. TheSAEDP is less stiff than a standard automotive glass window. In oneexample, the SAEDP comprises a passive material. In another example, theSAEDP includes an airbag. And in still another example, the SAEDPincludes a pneumatic pad. Optionally, the vehicle includes a display onwhich the representation of the outside environment may be presented tothe occupant. Optionally, the display is supported by the SAEDP and/orthe side beam. Optionally, the display is a flexible display.

In a similar fashion to the utilization of beams on the left side of thevehicle, in some embodiments, one or more beams may be used to helpprotect the occupant's right side. For example, in one embodiment, thevehicle includes a second side beam, located at eye level to the rightof the occupant, and a second camera configured to take video of theoutside environment to the right of the occupant. In this embodiment,the computer is further configured to generate a second representationof the outside environment to the right of the occupant.

Some aspects of this disclosure involve utilizing one or morenontransparent beams in order to help protect the occupant in a case ofa collision. The one or more beams are stiffer than automotive laminatedglass. In one example, a nontransparent beam comprises a high tensilesteel pipe and pressed material. In another example, a nontransparentbeam comprises an aluminum extruded shape.

In one embodiment, an autonomous on-road vehicle that weighs less than1500 kg without batteries includes a nontransparent beam, which isstiffer than automotive laminated glass, and is fixed at eye level infront of an occupant who sits in a front seat of the vehicle duringnormal driving. The vehicle also includes a camera configured to takevideo of the outside environment in front of the occupant, and acomputer configured to generate for the occupant, based on the video, arepresentation of the outside environment in front of the occupant ateye level. Optionally, the representation is presented to the occupantusing a display. In one example, the display is comprised in ahead-mounted display (HMD), and the vehicle further comprises acommunication system configured to transmit the representation to theHMD. In another example, the display is coupled to the inner side of thecompartment.

The following is a description of an embodiment of a vehicle in whichone or more nontransparent beams may be utilized to help protect thevehicle's front (and an occupant therein). In one embodiment, anautonomous on-road vehicle includes a nontransparent beam, a camera(e.g., the camera 142), and a computer (e.g., the computer 143).Optionally, the vehicle weighs less than 1500 kg without batteries.

The nontransparent beam, which is stiffer than automotive laminatedglass, is located, during normal driving, at eye level in front anoccupant who sits in a front seat of the vehicle. Optionally, thenontransparent beam is at least 60 cm long and at least 1 cm wide(herein “cm” refers to centimeters). In one example, the nontransparentbeam comprises a high tensile steel pipe and pressed material. Inanother example, the nontransparent beam comprises an aluminum extrudedshape.

The camera configured to take video of the outside environment in frontof the occupant. The computer is configured to generate for theoccupant, based on the video, a representation of the outsideenvironment in front of the occupant at eye level. Optionally, therepresentation is generated from the point of view of the occupant, andthe vehicle further includes a display configured to present therepresentation to the occupant. In one example, the display is comprisedin a head-mounted display (HMD), and the vehicle further comprises acommunication system configured to transmit the representation to theHMD. In another example, the display is coupled to the inner side of thecompartment.

In some embodiments, a plurality of beams, which may be similar to thenontransparent beam described above, may be located in the front of thevehicle. It is to be noted that the plurality of beams may notnecessarily all have the same dimensions or be made of the same exactmaterials. Optionally, the plurality of beams may be connected invarious ways in order to form a structure that can better resistcompartment deformation in the case of a collision. For example, in oneembodiment, the vehicle includes an additional nontransparent beam thatis not parallel to the nontransparent beam and crosses it to form an “X”shaped structure. This structure is illustrated in FIG. 30b where beams235 and 236 cross to form an “X” shape structure. Both of the beams 235and 236 are at eye level (i.e., at least a portion of each beam is atthe same height as a typical occupant's eyes). FIG. 30b illustrates oneexample of a configuration of a plurality of beams located in the frontof the vehicle.

Different designs of vehicles may benefit from utilizing one or morenontransparent beam as described above. In one embodiment, thenontransparent beam is embedded in a movable structure that changes itslocation with respect to the vehicle compartment. For example, thenontransparent beam may be part of a front door through which theoccupant may enter the vehicle. In another embodiment, thenontransparent beam is embedded in the vehicle compartment itself anddoes not change its location with respect to the rest of thecompartment. For example, the nontransparent beam may be placed in acompartment wall in front of the occupant.

In order to better protect the occupant from injury (e.g., during acollision), in some embodiments, the vehicle may comprise anontransparent Shock-Absorbing Energy Dissipation Padding (SAEDP)located at eye level between the nontransparent beam and the occupant.The SAEDP is less stiff than a standard automotive glass window. In oneexample, the SAEDP comprises a passive material. In another example, theSAEDP includes an airbag. And in still another example, the SAEDPincludes a pneumatic pad. Optionally, the vehicle includes a display onwhich the representation of the outside environment may be presented tothe occupant. Optionally, the display is coupled to the inner side ofthe compartment. Optionally, the display is a flexible display.

Various embodiments described herein include a processor and/or acomputer. For example, the autonomous-driving control system may beimplemented using a computer and generation of a representation of theoutside environment is done using a processor or a computer. Thefollowing are some examples of various types of computers and/orprocessors that may be utilized in some of the embodiments describedherein.

FIG. 31a and FIG. 31b are schematic illustrations of possibleembodiments for computers (400, 410) that are able to realize one ormore of the embodiments discussed herein. The computer (400, 410) may beimplemented in various ways, such as, but not limited to, a server, aclient, a personal computer, a network device, a handheld device (e.g.,a smartphone), and/or any other computer form capable of executing a setof computer instructions.

The computer 400 includes one or more of the following components:processor 401, memory 402, computer readable medium 403, user interface404, communication interface 405, and bus 406. In one example, theprocessor 401 may include one or more of the following components: ageneral-purpose processing device, a microprocessor, a centralprocessing unit, a complex instruction set computing (CISC)microprocessor, a reduced instruction set computing (RISC)microprocessor, a very long instruction word (VLIW) microprocessor, aspecial-purpose processing device, an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA), a digital signalprocessor (DSP), a distributed processing entity, and/or a networkprocessor. Continuing the example, the memory 402 may include one ormore of the following memory components: CPU cache, main memory,read-only memory (ROM), dynamic random access memory (DRAM) such assynchronous DRAM (SDRAM), flash memory, static random access memory(SRAM), and/or a data storage device. The processor 401 and the one ormore memory components may communicate with each other via a bus, suchas bus 406.

The computer 410 includes one or more of the following components:processor 411, memory 412, and communication interface 413. In oneexample, the processor 411 may include one or more of the followingcomponents: a general-purpose processing device, a microprocessor, acentral processing unit, a complex instruction set computing (CISC)microprocessor, a reduced instruction set computing (RISC)microprocessor, a very long instruction word (VLIW) microprocessor, aspecial-purpose processing device, an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA), a digital signalprocessor (DSP), a distributed processing entity, and/or a networkprocessor. Continuing the example, the memory 412 may include one ormore of the following memory components: CPU cache, main memory,read-only memory (ROM), dynamic random access memory (DRAM) such assynchronous DRAM (SDRAM), flash memory, static random access memory(SRAM), and/or a data storage device

Still continuing the examples, the communication interface (405,413) mayinclude one or more components for connecting to one or more of thefollowing: an inter-vehicle network, Ethernet, intranet, the Internet, afiber communication network, a wired communication network, and/or awireless communication network. Optionally, the communication interface(405,413) is used to connect with the network 408. Additionally oralternatively, the communication interface 405 may be used to connect toother networks and/or other communication interfaces. Still continuingthe example, the user interface 404 may include one or more of thefollowing components: (i) an image generation device, such as a videodisplay, an augmented reality system, a virtual reality system, and/or amixed reality system, (ii) an audio generation device, such as one ormore speakers, (iii) an input device, such as a keyboard, a mouse, anelectronic pen, a gesture based input device that may be active orpassive, and/or a brain-computer interface.

It is to be noted that when a processor (computer) is disclosed in oneembodiment, the scope of the embodiment is intended to also cover theuse of multiple processors (computers). Additionally, in someembodiments, a processor and/or computer disclosed in an embodiment maybe part of the vehicle, while in other embodiments, the processor and/orcomputer may be separate of the vehicle. For example, the processorand/or computer may be in a device carried by the occupant and/or remoteof the vehicle (e.g., a server).

As used herein, references to “one embodiment” (and its variations) meanthat the feature being referred to may be included in at least oneembodiment of the invention. Moreover, separate references to “oneembodiment”, “some embodiments”, “another embodiment”, “still anotherembodiment”, etc., may refer to the same embodiment, may illustratedifferent aspects of an embodiment, and/or may refer to differentembodiments.

Some embodiments may be described using the verb “indicating”, theadjective “indicative”, and/or using variations thereof. Herein,sentences in the form of “X is indicative of Y” mean that X includesinformation correlated with Y, up to the case where X equals Y.Additionally, sentences in the form of “provide/receive an indicationindicating whether X happened” refer herein to any indication method,including but not limited to: sending/receiving a signal when X happenedand not sending/receiving a signal when X did not happen, notsending/receiving a signal when X happened and sending/receiving asignal when X did not happen, and/or sending/receiving a first signalwhen X happened and sending/receiving a second signal X did not happen.

Herein, “most” of something is defined herein as above 51% of thesomething (including 100% of the something). A “portion” of somethingrefers herein to 0.1% to 100% of the something (including 100% of thesomething). Sentences of the form “a portion of an area” refer herein to0.1% to 100% percent of the area.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having”, or any other variation thereof, indicatean open claim language that does not exclude additional limitations. The“a” or “an” is employed to describe one or more, and the singular alsoincludes the plural unless it is obvious that it is meant otherwise.

Certain features of some of the embodiments, which may have been, forclarity, described in the context of separate embodiments, may also beprovided in various combinations in a single embodiment. Conversely,various features of some of the embodiments, which may have been, forbrevity, described in the context of a single embodiment, may also beprovided separately or in any suitable sub-combination.

Embodiments described in conjunction with specific examples arepresented by way of example, and not limitation. Moreover, it is evidentthat many alternatives, modifications, and variations will be apparentto those skilled in the art. It is to be understood that otherembodiments may be utilized and structural changes may be made withoutdeparting from the scope of the appended claims and their equivalents.

We claim:
 1. A system comprising: a light shading module integrated witha vehicle window; the light shading module is configured to be in atleast first and second states, wherein the Visible Light Transmittance(VLT) of the vehicle window in the first state is higher than the VLT inthe second state; a camera, fixed to the vehicle, configured to takevideo of the outside environment; a computer configured to generate avideo see-through (VST) based on the video; and a head-mounted display(HMD) configured to operate according to a first mode of operation whenan occupant inside the vehicle looks at the direction of the vehiclewindow and the light shading module is in the first state, and tooperate according to a second mode of operation when the occupant looksat the direction of the vehicle window and the light shading module isin the second state; wherein the total intensity of the VST light,emitted by the HMD and reaching the occupant's eyes, is higher in thesecond mode than in the first mode.
 2. The system of claim 1, whereinthe vehicle is an autonomous vehicle, and the light shading modulecovers more than half of the front windshield in the second state. 3.The system of claim 1, wherein in the first state the VLT is above 10%of ambient light entering through the window, in the second state theVLT is below 50% of ambient light entering through the window, and theVLT in the first state is higher than the VLT in the second state. 4.The system of claim 1, wherein in the first state the VLT is above 70%of ambient light entering through the window, in the second state theVLT is below 30% of ambient light entering through the window, and theVLT in the first state is higher than the VLT in the second state. 5.The system of claim 1, wherein the HMD comprises an optical see-throughcomponent and a display component; in the first mode the intensity oflight that reaches the occupant's eyes via the optical see-throughcomponent is higher than the intensity of light from the VST that isemitted by the display component and reaches the occupant's eyes; and inthe second mode, the intensity of light from the environment thatreaches the occupant's eyes via the optical see-through component islower than the intensity of light from the VST that is emitted by thedisplay component and reaches the occupant's eyes.
 6. The system ofclaim 5, wherein the perceived contrast of the optical see-throughcomponent is better when the light shading module is in the second statecompared to when the light shading module is in the first state.
 7. Thesystem of claim 1, wherein the computer is further configured not togenerate the VST when the HMD operates in the first mode.
 8. The systemof claim 1, wherein the light shading module is a movable physicalelement configured to reduce the intensity of the ambient light enteringinto the vehicle compartment through the vehicle window.
 9. The systemof claim 8, wherein the light shading module is unfurled from the insideof the compartment in order to block at least 50% of the ambient lightintensity.
 10. The system of claim 8, wherein the light shading moduleis unfurled from the outside of the compartment in order to block atleast 50% of the ambient light intensity.
 11. The system of claim 1,wherein the light shading module is a curtain.
 12. The system of claim1, wherein the vehicle window is made of a material able to change itstransparency properties.
 13. The system of claim 1, wherein the VSTrepresents the outside environment from a point of view of the occupantwhile looking at the outside environment through at least a portion ofthe vehicle window.
 14. The system of claim 1, further comprising anoccupant tracking module configured to calculate the point of view ofthe occupant based on measurements of a sensor.
 15. The system of claim14, wherein the computer is further configured to render the VST basedon data received from the occupant tracking module.
 16. The system ofclaim 15, wherein the HMD comprises an optical see-through component anda display component, and the display component is a three dimensional(3D) display configured to show the occupant the VST, such that point ofgaze of the occupant, while looking via the 3D display, is most of thetime beyond the location of the light shading module.
 17. The system ofclaim 15, wherein the HMD is an augmented reality device.